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ABSTRACT i The purpose of this experioental study was to develop and evaluate a pa^tticular node of 
cooputervext ended instruction (CEI) id\ introductory calculus. Prepared computer jirograns were used by 
students lt^ an otherwise conventional "talcolus class to investigate the concepts ^f functions, lioits and 
derivatives. The CZt was evaluated on tw disensions; concept a<.tainaent and general knowledge. , 
The r'esolts Indicated that the use of CEI can have a significant effect on the learning of basic concepts 
of introductory calculus. Male students seeaed Co benefit particularly fjroa the, CEI, while female 
students were not measurably affected by it. No significant effect on the learning of general knowledge 
In calculus was observed. However, there was some indication of the presence of a differential effect 
(interaction) of the CEI on students with varying level^of proficiency in algebra and trigonometry. 



* There is a growing awareness among educators 
that the vast capabilities of modern computers 
have brougJiti. us to the verge of a revolution in 
education that far exceeds anything that man has 
ever experienced. It is easy to see that the com- 
puter technology which now exists could readily ^ 
provide us with»a totally different instructional 
program in our schools. Some changes ha:ve already 
been implemented while* needed experimentation is 
continuing briskly in all areas bf instruction. / 

Mathematics educators have been interested in 
this computer revolution right from the start. In 
fact, most of the pioneering studies in computer 
aided learning wete done with mithematica studcnta 
at various levels. This interest has expanded ex- 
ponentially 80 that now many mathematics edu&ators 
feel that the computer will soon become a standard- 
tool throughout the mathematics curriculum. It is 
therefore of great importance that we direct our 
attention toward devising effective modes ff com- 
puter utilization in the classroom. ^ ^ 

In this paper I would like to describe a te- 
search study which concerned itself with the 
development, implementation and evaluation of a 
particular type of computer use in an introduc- 
tory calculus claas. 

Modes of Computer Use in' Mathematics Education 

Several modes of computer utilization have 
been suggested for mathematics programs in the 
schools and colleges. Three general categories 
that are often mentioned are Computer AaSisted 
Instruction (CAI)wComputer Managed Instruction 
(CMi) and ComputerExtended Instruction (CEI). A 
btief if somewhat oversimplified explanation will 
serve to distinguish these categories. 



In CAI, the computer more or less assume^ the 
role of the instructor. It may be used for the 
purpose of presenting a particular learning se- 
quence, providing drills ^nd exet^cises to the 
student, and testing diagnostically as well as 
for mastery (Suppes, 1968). Though the teather^is 
not replaced, many of his traditional functions / 
are altered and he is freed to concentrate otr ix/- 
dividual problems in his class. 

CMI, ort the other hand, provides a managerial » 
tool for the instructor. The function o^ftthe • 
computer in CMI Is ta assis|k the teacher and the 
student in planning instructional sequ^ces. 
Through repeated diagnostic evaluations and de- 
tailed record keeping handled by C^c; computer, a 
student is able to .receive indiviiiualized pre- 
scriptions, for learning tasks. 

CEI is a^ label which has befcn used to de- 
scribe the rol^ of computers as a supplementary 
teaching tool in a conventional classroom setting 
(Bitter, 1971; Dorn, 1970). Though the instruc- 
tion is not presented on the computer, CEI is 
intended to provide the teacher with instructional 
capabilities that' would not otherwise be possible. 
At the same time the student^ can be provided with 
a tool Zd aid him In mathematical explorations 
far beyond his usual ll^mits. In this mode, the 
computer becomes a type of laboratory device which 
can be used as an aid to the teacher In presenting 
material to the students or as a means of leading 
students to observe mathematical properties, to 
•discover mathematical relations, and to perceive 
mathematical concepts. ^ • * , 

It was this third category. Computer Extended 
Instructioh, Xthich was the fo£us of this research 
study. The specific objectives were to develop 
and evaluate a particular mode of CEI la a col- 
lege level introductory calculus course. 
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The use of CEI an incroductory calculus 
course has received atcencion by 'a nuaber of re- 
searchers in recent years. Alreaay nine years ago, 
Smith (1970) conducted an c^perioenc with honors . 
level calculus students in which the students were 
giveo^ special computer assignments to investigate 
the limit of functions. In 1968', Fiedler (1969) 
carried out a study to determine whether students 
develop a deeper grasp of some of the concepts of 
calculus through programming problems dealing' ^with ^ 
these concepts oa a computer. Later duxiiiigl^^ 
same year, BUter (1970a) conducted an expef$;?en- . 
tal calculus course in which students used a'Xime- 
sharing system to work special computer assign- 
ments. The objectixe was again to illustrate and 
fortify some o£. the bisic concepts of calculus. 
The experiment by Bitter was repeated with some 
revisions the following year and ^ formal educa- 
• tlonal study was carried out to assess the effec- 

*^iveness of this approach (Bitter,* 197-Ob) . 
Holoien (1970) conducted another study in the Fall 
of 1969 using sobe computer augmentation with stu- 
dents of Introductory calculus and a few years 
later Bell (1972) devised still another experiment 

^here certain computer projects were used to en-, 
hance the learning of calculus concepts. 
\ 

Though not all of these studies produced con- 
clusive results, the studies of Bitter, Holoien, 
and Bell seemed to give srtrong, indication that the 
use of CEI in introductory calculus provided an 
effective mean^ of improving ins^dmotlon. Statis- 
tically significant results were obtained by^each 
of these researchers favoring CEI groups over 
classes receiving only conventional instruction. 
It seemed also, transparent* in these results that 
the teaching of fundamental concepts of calculus, 
such as functions, limits, derivatives, etc., 
rather than techniques was most positively bene- 
. fitted with the use o^ CEI. 

A Special ^tode of CEI 

Although the above mentioned studies were all 
similar in their objectives, the nature of the 
CEI that was employed was quite different in each 
case. This resulted both from the difference in 
computing power available on the various campuses 
and the difference in the philosophy 6f how the . 
computer should be used. In all cases cited 
above, students were required to learn a simple 
computer programming language and to *write some 
of their own programs. 

Because of certain problems encountered whel^ 
students must write their own programs' to' illus- 
trate a mathematical concept, it was decided for 
this study to develop several programs which 
would be readily available and usable l>y the* stu- 
dents, including those with little or no know- 
^ ledge of computer programming. Consequently, 
programs were devised by the experimentor to 
illu8tr:ate functions, graphs, limits of functions 
and derivatives. The programs were general in* 
nature, requiring the student to supply the func- 
tion to be investigated and the necessary para- ^ 
meters (e.g., a value where the limit of the 
function was to be taken). The programs were 
stored on disk files on a large multi-programmed 



computer system (CDC 6600/6400) and were accessed 
through the submission of four or five control 
cards in the batch process ingmode. Turnaround 
time was minimal- on this sysjPi — usually only a 
few minutes — providing the students with almost 
immediate .results and the opportunity to freely 
experiment with varying inpu^ parameters. 

The type of computer extended calculus des- 
cribed above was labeled the "PCP mode of CEI" 
(Prepared Computer -Programs mqde) to distinguish 
,it from othe^ forms of CEI. (At the tim^ the 
study5waF>a^, this form of computer use was ^ , 
- ^udged'to ^e^npst a^ro^jriat:^ forj tfte ^rtifculSr 
students i.nvolved. Similar types of CEI could of 
course be devised on time-sharing systdas using 
teleprinters as'I/0 devices.) 

^^Ali' the computer augmentation was accomplished 
thrbugh special supplementary assignments which 
directed the students to investigate certain phe- 
nomena regarding various* mathematical functions. 
The emphasis was on teaching the underlying con- 
cepts' of calculus and not on tea9hing problem- 
solving ox techniques^. • 

Experimental Desij^a • . 

To assess the effectiveness of using this 
special mode of CE«X in introductory calculus, an 
experimental' setting was created and several for- 
mal hypotheses were formulated and tested^ The , ^ 
design for the experiment was the (non-equivalent) 
pretest-poijttest control group design (Campbell 
and StanleVt' 1963). Twg^retests were gjven to 
the subjects to control -for initial differences in 
preparation in precalculus mathematics, while two 
posttests served as se^Jarate Pleasures of effec- 
tiveness in the study. 
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The students involved- in this experiment -were 
selected from introductory cdlculu^ classes in the 
Fall semester o5 1972 at the University of Texas 
at Austin. 

Two lecture classes taught by senior faculty 
.members were arbitrarily selected to participate 
"^'in the experiment. ' Each of these classe^^as 
' subdivided Into four dispussion sections, headed 
by two teaching assfttonts. Thus a total of two 
professors and ,four tieaching assistants (besides 
fhe experimentor) were indirectly Involved in the 
study. Complete test data were available for 
subsequent analysis from 84 students in these 
classes, and from this group certain subsamples . 
were defitied. 

In order to evaluate the effectiveness of this 
special mode of CEI in introductory^ ca:}.culus, the 
sample was partitioned into two gro^ips ' Four out 
of the eight discussion sections 'involved in the 
study were selected to comprise the experiment^al * 
or CEI group, while the retaaindex was designated 
as the cpntrol or Non-CEI group. ,This assignment 
of intact classes to the CEI and Non~C^X groups 
was random, ^ but subject to certain constraints 
which iissured a crossing of professors', teaching 
assiBtairt^s and class time with the experimenfal^ 
variable, BotV-^^oups wer* provided supplementary 
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assignments', but the computer augmentation was 
confined strictly to the CEI group. ^ 

Several sp^^cific questions (which are given in 
the next sectj^on together with their responses) ^ 
were considered during the evaluative phase of 
this study. (The formal, hypotheses^ relevant t6 
these questlons^ill not be stated here.) Statis- 
tical tests were made using two separate crite- 
rion measures.' One of these was the author- 
produced Calculus-Concepts '^est, a test of the 
basic concepts of introductory calculus, while the 
second criterion was the Calculus-General Test, a 
standardized test of general knowledge in intro- 
ductory calculus. Two standardized precalculus 
tests, one in algebra and one in trigonometry, 
served as covarlates in the study. The general 
technique of analysis used was the multiple linear * 
regression analysis. 

Conclv(sions 

The three fundamental questions that were 
asked in this atudy and their responses which are. 
based on the statist^ical analyses follow. 

Question 1 . Does the PCP oode of CEI, when 
used in an introductory college calculus class, 
have a po/sitive effect on achievement and concept 
attainment? 

' The Statistical tests of the main hypothesis 
which pertained to this question provided strong 
indications that an affirmative answer is justi- 
fied when 'the focus is on concept attainment, 
especially for the male students. When ^consider- 
ing 5:hose stjadents in the sample who showed active 
patticipation in the experiment, a nearlv signifi- 
cant- (o - .059) difference in adjusted mbans was 
observed on the Calculus-Concepts Test favoring 
the CEI group over the Non-CEI group. F<ir the 
subsample consisting of only male stud^ents, this , 
•difference was clearly significant (a - .016), 
again favoring the CEI group. No significant 
difference was apparent for the female subsample,* ^, 

vbn the other hand, when achiev'ement in calcu- 
lus was measured by a test of general knowledge 
(the Calculus-General Test), no statistically sig- 
nificant differences between the CEI and, Non-CEI 
groups w^re apparent . It should be kepC in mind, 
however, 'that the CEI was not aimed at teaching 
gener^al techniques^ but was primarily designed to 
elucidate the basic concepts of calculi^^s. 
- ^ i 
The (Analysis pertinent to Question/j. revealed 
, ^ therefore that the use "of C^I can be an effective 
V supplementary tool in teaching the basic concepts 
^ of calculys. However, with the particular objec- 
• tives which were used in this study, the CEI 
seems to be of questionable value in bringing 
about improvement in the general ^ttephniques of 
.calculus, although there are ^ppar^itly no dele- 
terious effects. This is in pricise agreement 
with this fii^ding of Bell (1972)., Holoien's ( 1970) 
results seem. also to point in this direction since 
he found that certain concepts, suth as the limit ^ 
of a function and the evaluation of a function* 
seemed to benefit more directly by the computer 
augmentation than other topi<is. Bitter (1970b) 
found in his experiment that even on a general 



tfst, the CEI group outperformed the Non-CEI .group. 
•Nc separate test of the effects on concept attain- 
ment was made by Bitter. ^ 

It will be obseVved that since each of these 
studies utili.zed different modes of CEI in calcu- 
lus, they cannot be c6nsidered as replitations. 
Nevertheless, the general support for the i^se of 
.CEI in calculus.* particularly toward "teaching the 
basic concepts functions, limi;:s, derivatives, 
and so forth, seems to point toward the. fact that 
the computer can be used effectively to enhance 
calculus instructioi!. Therefore serious thought 
should be given toward implementing CE5 in.iot^o-. 
ductory calculus in our colleges and universities. 

Question 2 ) Is there a differential effect 
from the use of the PCP mode of -CEI on students 
of varying levels of ability in precalculus mathc* 
ma t Ic s ? . ' ' ^ 

The results of th^ ana^^ses dealing with this* 
question showed that when the criterion- Is concept 
attainment," there' is no reason to siispect any 
interact^fiUJQf.j:^ceapcwaiC>lttuprior achievement in 
precalculif^ mathematics. However, when a test of 
general knowledge was the criierion, there was * 
some indication that students with high trigonome- 

'try pretest scot*es benefitted mo»re from the CEI in 
calculus than those with low trigoryjmetry scores, 

"while the reverse seemed to be true for students 
with high algebra pretest scores. In the 'statis- 
tical tests, this differential ef*fect of CEI on 
students with varying levels of precalculus 
achievement" was observed to be close to Signifi- 
cant, with an a-level of .084. Thus the answer 
miay well be affirmative when a general knowledge 
of calculus is of primiary concern. 

, * <• 

Figure 1 shows the regression planes for the 
CEI and Non-CEI groups and indicates the region 
where the expected score of the CEI group exceeds 
that of' the Non-CEI group for different combina- 
tions of trigonometry and algebra pretest scores^ 
From the equations of the regression planes, it 
can be shown that the predicted score for a^mtmber 
o.t the CEI group is greater than that for a member 
' of the Non-CEI group vhenever the linear inequal- 
ity 35A - 29T < 348 is satisfied, where A and T 
are the Algebra and Trigonometry pretest scores. 

If' this result regarding the effect of CEI on 
the acquisition of general knowledge in calculus 
can be substantiated in further research,' it may 
be advisable to use different strategies of In- 
struction — some with CEI, others withoqt — for 
students of varying levels of prior achievement in 
trigonometry and algebra. Considerably more r^e- 
search is necessary, however, before any such 
conclusions are warranted. " ^ \ * 

Question 3 . Is fihe effect of using the PCP 
n^de of CEI d^^ferent betweep the yAXj and female 
students? *, 

In regard to this question, "the.etatistical- 
tests provided some evidence that there is a dif- 
ference in the, ef fetftiveness of using CEI J)etween 
the male and female students when the criterion .is 
concept attainment. The test of the main hypothe- 
sis u8^.ng only male subjects tevealed that the 



ERIC 




(T 



Trigonoactry Pretcat 



Figuro 1, ' 
Pectcsaion Plajica for CKI and Hon-CZT Groupa 



adjusted means on the Calculus-Concept^'Xcst v^re 
significantly* (a ,016) higher for the CEI group^ 
than for the Non-CEI gtoup. A similar test re- 
stricted to the- fQmale subsample showed no' signifi- 
cant differencb. On'^he other hand, when the 
Calculus-General Test was used as criterion, there . 
were nonsignificant differences between the CEI and 
Nt)n*CEI groups for either the male or female sub- 
samples. 

The fact that the male subjects apparently re- 
sponded more favorably to the C£I than the femile 
subjects substantiates a similar observation by 
Bitter (1970b) who surmized that this difference 
may be the' result of a difference in attitude ' 
toward mechanical devices by these two groups. It 
may also be possible that the female students tend, 
to be more apprehensive and fearful «than the male 
students when confronted with new and unfamiliar 
situations. The experimentation in. computer ex- 
tended instruction certainly provided some uncer- 
tainties for the students throughout the semester, 
and it may, be this factor that destroyed the poten- 
tial effectiveness of the CEI for the, female 
students. 

Possible implications of this result are that 
different modes of instruction may be appropriate 
for male and female subjects in order to achieve 
optimal IjBarning in calculus. Much more research 
Is needed before such action is contemplated, for 
it may well be that there is a need to provide the 
female students with more opportunities of dealing 
.with mechanical devices or of exposing them to 
more^niquc and unfamiliar situations at 2y\ earlier 
age ko as to eliminate any unusual misgivii^gs they 
may nAve toward computers. 



Limitations^ 

Several factor^" which may have affected the 
outcomes of &his experiment should b6 considered 
when evaluating the results. Any conclusions 
based on this study must be tempered with these 
limitations in mind. , • . . 

1. ^ The subjects who were included in the ex- 
periment were all students wha had enrolled in 
the introductory calculus classes taught by two- 
cooperating pro fes SOPS at the University of Texa%. 
Although the assignment of subjects to the exfteri- 
mental and control groups involved a process of / 
randomization, the original selection of the 
sample^irom the total population of introductory 
calculus students at the University of ^exas at 
Austin was not random. Therefore, any generaliza- 
tions beyond this population at the University of* 
Texas will reqiiire even greater care. 

2. The homework assignments which served as^ 
the vehicle for the CEI were supplementary and 
vcre given fn addition to convent ion^felya^signprents 
relating to Che class lectures. SinoiC^he^pple- 
mentary assignments ware given to th^ students* 
through their teaching assistants while the regu- 
lar assignments originated .directly from their 
professors, there ^as a tendency for the students 
to place more weight on the regular assignment's. 
This fact se6ms to have significantly reduced the 
number of active participants in the experiment 
and may in addition have diminished the impact of 
the comjSutcr augmentation. ^ 

3. Although the problems which were investi- 
gated in the supplementary assignments inVblvcd 
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standard topics from introductory calculus, the 
timing of I the assignments was not precisely co- 
ordin^ated with the lecture presentations of these 
topics. Thus the effect of the CEI may conceiv- 
ably l)e increased with a closer conformity of the 
assignments with the classroom presentations. 

4. Independent inyestiigations apart from the 
assigned problems using the prepared 'computer 
programs were also encouraged, but the rapid pace 
that is typical in a beginning calculus course 
and the considerable demands placed on the stu- 
dents due to the dual sets of homework assignments 
seemed d^rimental to the realization^ of the goal 
of individual experimentation. Closer coordina- 
-Erion of the CEI with both the classroom presenta- 
tions and the regular assignments may also lead to 
more spontaneous computer experimentation, and may 
enhajnce the effectiveness of the CEI. 

5. The mode of CeS^ which was used in this ex- 
periment involved the itud-ent use of prepared pro- 
grams dealing with certiain basic concepts of in- 
troductory calculus. . Student access to the 
computer was ^aia^d at one of several remote Job- ' 
entry terminals, using a "isatch-processing'' method 
where turnaround time was usually between five and 
ten minutes. At installations where computer 
turnaround time is substantially longer than this, 
the positive efiects of this type -of CZl may 
seriously reduced. Howler , -ainor modifications 
are easily possible which would provide access to 
calculus programs of this nature via on-line 
teleprinter terminals or CRT terminals when »8uch 
terminals are readily availabrle. 

6* the fact that no positive conclusions 
could be made regarding the effectiveness of the 
CEI relative to learning the general techniques of ^ 
calculus' is probably due tp the fact that the sup- 
plementary assignments and the computer programs 
were specifically intended to strengthen the under- 
standing of sdide of the tiasic concepts of calculus 
and not the techniques. This objective wa? appar- 
ently accomplished without causing any deleterious 
effects on the learning of techni<|^ues. Therefore . 
it should not be concluded that^tl^e CEI cann9t be 
used also as a tool for enhancing the learning of 
otber aspects of introductory calculus. The type 
of assignments as well as the mode oi computer use 
will surety need to be altered to bring about pos- 
it^lve results in this direction^ 

Recommend at ions * 

The following recommendations are suggested in 
consequence o£ the experiences gained through the 
research study in computer extended calculus con- 
ducted at the University of Texas at Austin in the 
fall of. 1972. 

1. When CEI is used in calculus, it should.be 
carefully integrated into the regular instruction- 
al program, with conventional as well as computer 
oriented assignments originating from the instruc- 
tor who^ is primarily responsible for the course. 

Z> Although the, use of "batch processing" in 
CEI has the advantage of providing the user with 
printed resVlt^ including long lists and extensive.- 
graphs, the u«e of on-line teleprinter and CRT , 



terminals should be carefully explored. ^ Whenever 
the' turi>around time of batch-grocessed programs is 
excessive, these* alternate mean? cdtoputer com- 
munication seem preferable. Further research is 
needed t° explore the possibilities of such use 
and toa:9«es§ their effectiveness. * ^ 

3. Because students who are unfamiliar with 
computers often have a considerable fear of them 
or of the peripheral equipment associated with 
them, it is advisable to provide some clos^ guid- 
ance in the use of any equipment which is utilized 
in conjunction CrltlTthe CKi programs. Carefully 
written instructions are important but often in- 
adequate to dispel initial fears, so personal 
instruction should be provided at the starts. -A 
computer consultant who is familiar with the CEI 
programs should also be available for additional 
help throughout the du;ration of the course. 

4. The use of' the prepared programs should be 
as simple as possible so as to provide the student 
with a readily Usable means of investigating math- 
ematical concepts without first requiring him to 
learn a new complex skill. ^ 

5. Additional programs illustrating basic 
mathenlatical ideas in various courses should be 

^developed and research assessing their effective- 
ness as part of the instructional program should 
be conducted. 

6. Further invest.igation into the possible 

t ptesence of an interaction between the use of CEI 
in calculus and the prior ability of students in- • 
algebra and trigonometry is necessary. 

7. Concerns for the differential effective- 
ness of CEI on male and female students need fur- 
ther research. 

Concluding Statement 

The general purpose of this study was to 
' develop a particular mode^of computer extended in- 
. struction in introductory c^JLculus and to assess 
its effectiveness in bringing about learning. ^ The 
results of the resear<;h' showed that the PCP mode 
of CEI which was (developed for us* at the Univer- 
sity of Texas at Austin can be effective in 
teaching basic concepts and should be explored 
more fuUy as an aid in calc{^l,us instruction. 
This generally confirms the earlier results of 
similar stucUes'by Holoien (1970), Bitter (1970b) 
and Bell (1972) , who also found CEI to be a use- 
ful tool in calculusT instruotion.' ^ . 

It is true that the computers which fljre'^vailable 
at different schools vary widely ar^i|th4t computer 
access by stuflents is possible in ttAtiy WaysJ' Ex- 
perimentation at the local level w£tfc. severgl 
modes of CEI in various areas of matbemaiiiical con- 
tent is therefore strongly recommended prior to a 
wholesale commitment to one f orm t>r anothet.^ 
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.\BSTRAC.T. Goal Programming (a generalized extension of linear prograjnming) is a method of allocating 
resources while considering multiple objectives or goals. The brief history of goal prograraming <;ill be 
explained with reference to its development from linear progranmiing. The basics of problem formulation 
*wlll be discussed f^r educational settings. The primary requirement of problem formulation is an orJiifel 
ranking of the goals. These goals may represent inconmenfeurable quantities. Likewise, tlia goals may be 
mutually conflicting. Examples of educational applications will be presented and discussed with respect to 
linear programming solutions to the same or similar problems. 



Coal programming is both a modification and 
extension of linear programming (Lee, i972). Lin- 
ear progi;^mmiTig is the name given to the operations 
research technique wherein a mathematical model of 
n problem situation is formulated to con(tain a 
linear objective function and constraints. Through 
the use»'of an iterative algorithm known/ as' the ^ 
i^^implex technique or a modification the/teof, the ' 
Objective function and i^8 components^ /homogeneous 
choice variables, are optimized (maximized or 
minimized) subject to the constraints^ ulimited re- 
sources or restrictions) , stated in thf model, 
constraints rept,esent relationships between the*^ 
choice variables and are stated *as liliear ine- 
qualities and/or equalities. Betause of the 
.vailability of computer programs for performing 
Lhe mathematica^l calculations necfessary to the 
isolation of complex linear programmi/ng *pH:oblems* 
it is not the computation but rather the model 
formulation that is the chief concejrn of * the 
;^robeim-solver . 

The goal progrflntming approach </as originally 
formulated by Charnes and Cooper 0.955) and named 
^n Volume I of Management Models and Industrial 
Applications of~Lin.ear Programming (1961). The 
method presented used a weighted Objective func- 
' tion.* Ijirl (1965) presented an ilgorithm for 
goal programming in which the objective function 
is prioritized rather than weighted. This causes 
all resources in a problem to ba applied to the 
first goal until it is satisfied and then the 
second goal until it is satisfied and so on. By 
1968 Charnes and Cooper et. al./, had made appli- 
cations in the fields of media /planning and man- 
power planning. In the last six years the number 
o/ articj^s concerning goal prpgramming has in- 
creased greatly, and one book^/in particular (J^ee, 
^972) provides an excellent introduction to the , 
area. Lee presented an algorithm for goal pro-« 
^ramming chat is a modif icatlJon of the standard 
linear programming simplex mdthod. The computa- 
tional basis of the examples /discussed here 



depend on ^Lee's al^orithnv and his Fortran program. 
His progratn has also been adapte-* for time-sharing 
on the H-P (access) system at the ^Jniversity of 
Iowa . 

How Goal Programming • ^ . 
and 

Linear Programming are Alike 

\. Both require the formujlation of a model for 

transform'ing a real-world decision problem into 
^ a prescribed format. 

2. Both are concerned with ^oal or^ objective 
achievement . 

3. *^ Both have thfe following mot^.el characteristics: 

*a* The optimization of the* objective function 
subject to ^ set of constraints or limited . 
•Tirfources. 

b. Variables must have the property of non- 
* negativity. * 
' c. Constraints and objective function are 

Mnear,. i.e. all variables are ^o be jthe 
^ first- power or "ot the first, order".. No 

quadratic or higher order relationships may 
be incl.udcd in the model fc^rmulation. 
A. Both represent a systematic attempt toward 

rationality in decision-making. 
5. Both are adaptable to analyzing decisions. 

When a computer program is available to perform 
the necessary calculations, the educational 
m^naget c^n easily "try-out** different formula- 
tions of the problem model. »Ie can, for exam- 
ple, observe the effects upon the solution of 
changii>g the coefficients of constraint 
variables. 

» How Goal Programming f 

is ^ 

Unlike Linear Programming 

1. Goals — The goals of a particular problem are 
modeled as constraints althou;>h they may be 
statements (written In the same format as 



283 



constraints) which are not restrictive or 
descriptive of limited resoui'ces but positive 
in nature, representing a desirable condition. 
These constraint/goals will be hereafter . 
referred to simply as goals. Since it is not 
usually necessary that .each goal in the model 
be achieved exactly, goal programming allows, 
for the likelihood that goals in a real- 
world problem may be conflicting. The devia- 
. tlonal variables preserve th^ equality of 
each goal when combinations of goals are con- 
flicting. AlthougR the choice variables 
within goal;s must be consistent as to units 
of measure, goals may represent incommen^r- 
able quantities. 

Deviational variables — A feature of gpal 
programming models noj; found in linear pro- 
gramming models is the use of deviational 
•variables. These variables enable all goals 
to \>e stated as equalities. Of primary con- 
cern are the variables known as slack and sur- 
plus, deviational variti'bles. Aft&r all goals 
whTch are to be incorporated in th^ model are 
identified, each must be assigned slack and/or 
surplus variables. 

1. Objective function — In goal programming the 
objective function usually contains no ch'sice 
variables, but rather is made up of the devia- 
tional variables contained within the goals. 
When multi^e goals are thus represented in 
the objective function, i^ is said to be 
multidimensional. Because the optimal solu- 
ij.on would be the one in. vhich the sum of the 
deviations from goals is^minimal, the objec- 
* tive function is always minimized. In order 
that goals be achieved according to their 

^ importance, the deviational variables in the 
objective function must be prioritized accord- 
ing to the ordinal ranking of goals or the 
importance of .each goal to the manager. The 
same priority level may be assigned to one '6r 
mo^e deviational variables. Deviational vari- 
ables on the same priority' level may be 
weighted; it is perhaps most desirable to ^ 
weight such variables wl^en ^it makes clearer 
that the "cost",^^f underachievement of a par- 

t ticular*goal great tQr ;- ; !^ ;^. lesser than another 
of equal import anc'6;|*«;The xinTts of measure of 
the deviational vaS^Mes wi'thin the objective' 
function fiay be nonhomogeneous, e.g., repre- 
senting dollars and ^weeks, rather than one 
type of unit. ♦ 



Limitations of Goal Programming 

CI * 

Lee (1972) indicates four limitations of goal, 
programming as: (I) proportionality, (2) additi- 
vtty, (3), divisibility and (4) deterministic. Pro- 
portionality,^ as a limitation, means^ that the lin- 
CiiT relationships in the probletn model must be 
proportionol.^'Additivity* indicates that the 
activiti(^s, expressed in the objective function 
and goals must be additive in order to ensure 
linearity. Divisibility means that the values for 
decision variables in the optimum^ soluffion of d 
-'oal programming problem can be non integral. Re- 
cently, a study on integer goal programming has 
appeared (Ke'own and -Lee, 1975), therefore it is not 
i>nticipated that the application of goal program-' 
ingwill long be limited to a feasible . solution Set 



of positive real numbers. The deterministic nature 
of the goal programming model means that model 
coefficients must constants. In this sense, the 
goal programming procedures is not better than most 
oth.er rational procedures which requir^ a "snap- 
shot" of a continually changing uorld. , Again, 
there is reason to believe that this J-imitation may 
be eliminated or at least reduced, in light of re- 
cent w6rk showing that constraints r.uiy have varia- 
ble limits (Sween.ey and Williams, ]074) or repre- 
sent a unique probability distribution (Continl, 
1968). • 

Despite its present imitations, goal program- 
^g is believed to b^ applicable to a wide range of 
educational problems. 

Educationaf Applications 
of Goal Programming 

As an extension oj linear programming', it is 
assumed that goal programming can rqadily be adapt- 
ed to, the solution of educational projplems pre- \ 
viously utilizing linear programming techniques,-^, 
but it appears that a greater value of goal pro- 
graTirning lies in its facility for providing a nio^e 
realistic model of the decision environment than 
has previously been possible with linear programm- 
ing. * 

« 

Linear programming has been utilized in the 
solution of such educational problems as: 

1. minimizing travel distance in busing for 
integration TOntjes, 1971). 

2. maxlmi2i>ng the dtstrict-wide assignment of 
teachers^Berfcie,. 1972). 

3. optimizing various aspects of a foundation type 
of state aid program (Matzke, 1971). 

4. determiniiig university faculty salaried 
(Hartley, 11973). 

5. designing alternative forms of salary schedules 
for public school teachers below the ui\iversity 
level (Bruno, 1969,1970). 

Goal progrf^mming models have been created for 
the solutions tt) such educational problems 'as: 

1. allocation of resources in institutions qff 
higher learning (Lee, 1972).» 
• 2. determining^ a job factor comj/ensation plan in a 
publ*ic school setting (Gunderson, 1975). 

While applications of goal programming in edu- 
cation have been rel^ti^^ly scarce, there is 
, reason to believe that it will prove to be a valu- 
able decision-making aid to the school administra- 
tor once programs are widely available <Gunderson, 
1975). X 

• 

Model Formulation 

* . C . . 

The basfi: linear programming problem is 



5^ line 
^s foil 

^Optimizfe 2 - j^^c^x^ 
such tjiat; j.i^ij'^j^ ^'i 



and X. ^0 

wherp , b and c. are arbitrary constants. 
% 

The basic ^oal programming problem is formula-'* 
ted as follows: 

Min. ?. = (d^" + dp. 



and 



such that Ax + Id" - Id^ 
and X, d", d"^ ^ 0 



whbre A Is a_n ^ n matrix, I is a n x n iden- 
tity matrix and B Is a n component column vectdr. 

As in 'any model formulation the foUowing 
steps should be» taken: / ' 

t.^ define the variables and constraints. , 
43. formulate the constraint equations. * 
3. develop the objective -function. ' 



similarly for medium group (T^^), small group (Tg^ 



and Individual instruction (Tj), this may be 
expressed by: , 



250 



where d *is the number of minutes per week less 
1 y 

than 230 tliat '^^ch student spends in algebra and 

is the number of minutes per week more t;,han 250 
each student spends in algebra. ' 

Condition 2, Due to limited spate for large 
classes to meet, we would like to schddule algebra 
studepts for not more than 60 minute? per week of 
larfee group instruction. This is expressed by: 

' T, -d^"^ = 60 

I. 2 « ^ 

where d is the number of minutes per week over 60 
^ ■* • 

schedules for large group instruction In algebra. 



Summary 

• Goal programming is an extension and modifi- 
cation of linear programming which allows the 
educational manager to more closely simulate real- 
life situations. Both l^lnear and goal programming 
are optimization techniques which lend thenjselves 
to increasing the rationality of decision-making I 
The foremost value of goal programming is in its 
facility for -solving problems with hierarchically 
arranged, conflicting goals. While ch^e are 
presently certain limitations of goal programming 
which may slightly narrow the scope of its feasi- 
ble* applications, it is believecf thatlits poten- 
tial for educational problem solving ^ vast. 

, Examples of Educational Problems 

and ' , ^ 

Goal Proj^ramming Solutions 

In the following three sections, e:tamples of 
f>oal programuiing applications to educational prob- 
lems are given. The reader will find that, while 
all three examples are^simplif led , the complexity 
of the given model f orraiilations increases sub- 
stantiall5^ from one model to the next. 

I . Scheduling Instruction 

Problem 

In a high school, 60 students are enrolled in 
algebra. Those students can be taught through 
large group instruction (all 60 in a class), 
"ledlum sizecf group instruction (30 to a class), 
small group instruction (15^ ta a class) or indi- 
vidual instruction. We need to decide how muph 
time they shoifid spend in each type of instruc- 
tion, siibjcfct to certain conditions: 

Condition 1. Regulations require that each 
student spend at least 250 minutes per week in 
algebra class or individual instruction. But, 
we also want to avoid .having students spend more 
than 250 minutes per week in algebra. "Letting T^^ 
stand for the number*of minutes per week each 
student j^ill spend in large group instruction and 



Condi tipn 3. We would like to schedule each stu- 
dent for at least AO minutes per week of small 
group' J.nstruct I^n. This is expressed by: 



AO 



where d^ is the numbe'r of* minutes per week less 

than 20 that each student is schedul^^d Into small 
group instruction. • 



Condition A. We wo 



uid 1 



ike each student to have at 



least 10 minutes per/ week of individuals instruction. 
This is expressed by: 

\ t I Tj + d^" = 10 

where d^ is the number of minutes per week less 

than 10 that students spend in individual 
instruction. ; ' • , 

Conditioir 5. - We would like to limit the ^moynt of 
teacher time used to teach algebra t<3 l^^OiO minutes 
p'er week. Tliis is expressed by? 

T, + 2T„ + ATp + 60T^ - d."*" = 1070 
L _^ M b Id 

where d^ is the teacher ^ime used in excess of 
1,070 minute^ per week. 

, Priorities 

*^ Now priorities must be established for. the con-' 
ditions - actually for the deviation variables, the 
d*s. Let us place the highest priority on each 
student having at least 250 minutes ^f instruction 
^ per week. This is expressed as P^^d^^ . Let us say 

that our second priority is to use not more than 
1 ,070 minutes' of _j_eAcher time per week. This is 
expressed as' p2d- . ^Similarly, priorities are , • 
established for cl/e other deviation variables . 
these made up ou^ object function vnTrcKis 
expressed by: 



Minimize - P^d^ 



+ V3" 



S85 



12 



Model 



•Table 1 



Thus the. goal progranning model fo? this 
problem is: 



STUDENTS 



11 



'2"5 



250 



+ d. 



^1 ^' ^ 



Solvin>; the aJ>ove model yields the 
following results; 



« 40 
• 10 

1070 



Type of''tnst « 

/^large' Group (T^^) 
::ediun Groap (T^) 
Small Group (T^O 

Individual (T^) 



Min/WeeH 
60 

^n55 
• 10 



Vith that solution, student^ spend exactly 2^0 
. minutes per week in algebra aijd exactly 1070 min^ 
^utes per week of teacher time is used. Thus all 
iWDnditions except the one having to do with the 
mljIUJtes per week for small group instruction are 
met. Only '25 minutes per week are aJllcfcated to 
small group instruction rather than thG«40 ;/e 
.Ranted. That, however, was our lowest priority. 

2' Busing • . • » * 

* This^^example will deal with the busing of 
scudentfi to achieve specified percentages' (or 
better a range of percentages)* of students 
In schools by group^ (such as race, sex, voca- 
tional * interest , etc.). There hap been a great 
deal of work done on this problem using a linear 
approach (Stimspn and Thompson, 1974). Here a 
goal programming forjnulation of a busing problem 
and the solution for a sample problem will be 
■ Presented . '* .* • 

The basic problem can be viewe<l a-s having two 
requirements. The first is to. achfeve a speci- 
fied percentage range composition by group^ The 
other is to minimize Cransportation costs via 
minimization o6 total ^busing distance, 

* » The sample problem will be .constructed as 
follows. We will assume a community with three 
scho^djj and three corresponding tracts that pro- 
vide' studi^nts for the three schools respectively. 

* The ^tuS^ht i?opulation by groqp, the school 'capa- 
cities, and the busing distances (using an aver- 
age distance) are summarized in Tables 1 and 2. 





Tract \ 


Group 1 


Group 2 


School Capacity 


+ 


1 

2 
3 

< 


450 
600 
50 


225 
0 

700 


750 
1000 
650 

♦ 




Totals 


1100 


925 


2400 



'Total Student Population: 2025 
' Table 2 
Distances to Schools (in miles), 



\ 



Tract \ 


1 


2 


3 




1 


1.2 




1.5 


3.3 




2 


2.6 


4.0 


5.5 




3 


0.7 


l.l 


2.8 





' A linear programming fonnulation«of this prob- 
lem could only consider one objective ^ busing 
distahc^^r percentages of students by grouy^s^ The 
goal programming formulation can consider both ob- 
jectives. In' the problem four priorities will be 
used. Priority one will be to have all "students 
assigned to a school. Priority 2 will be to have 
no school assigned more Students than its capacity 
will allow. Priority 3^ will be tp achieve a stu-, 
dent composition such that each S^oup f^Hs within 
a range of 40!!: to 60% of the total school popula- 
tion.^ And finally, priority four will j'be to mini- 
mize the total busing distance. j 4 ^ • 

The problem can be summarized as follows where 

' ijk * the number of students from tract i 
^ school j from group k. ^ 



•» ScAool 




* 








+ 




750 






+ 




1000 






+ 




650 





These constraints force students to be assigned 
to all schools with no school filled beyond its 
student capacity. 



Tract 

and 



2 "2jr*V 



and 

■ rx 



2j2 ^ 'l^ 



3 rXjjj+dg 
and 



450 

225 
600 

0 
50 



28b 




•These constraints force all ti*e-^udents 
to be assigned to a school. ^ 



Ratios for schoc^l: 



1 



-0-^^^ill'^°-^'^12*^10 



and 

-o.erx 



121 



and 
'.OilX 



121 



°-'^"l22^12 



and 
O.AIX 



^3, -0.61X^3, +d,3 



These constraints establish the 40% to 601 
Vange for each group of students. 



Distance 



+ 2.6rx 



21k' 



32k 



3800 



This' constraint forces the total distance, 
based beyond 3900 miles, to be minimized. ,The ' 
vglue of 3800 miles was obtaiiied as i he solution 
,to a linear programn\ing transportation problem 
for the given data with the objective being to 
'".inimize total distance bused. Therefore for 
this goal programming problem, 3800 miles repre- 
sents an idea) minimum distance. 



Now according to stated priorities the ob- 
jective 'function is 

3 

Minimize E 



''I'iVi'"''^ i-i'^i'/ 



15 

^3 wo-'i ^-Vie 

The solution is sumfaarizod in Table 3; it 
^hows^the numl^r of , students assigned from .each 
tract and grou^ to each school. 



Tabl'e 3 





1 


2 


3 


School Group 


0 
0 


A 50 
0 


0 

300 


■ ^ 2 


A50 
225 


0 
*0 


50 
175 




0 
0 


-150 
0 


0 

225 


Totals 


675 


600 


750 



Croup School 
Totals Totals 

300 ^ 



500 
^00 



900 



As can* be seen priority 1 was net completely 
with all students assigned to a school. Priority 2 
(filling all schools) was not net. There was a 
deviation of 375, but that i« exactly the amount of, 
excess capacity for th'fe three schools. The desire 
to meet the A02: to 60% compositior. for each group* 
was met exactly in schools one ^ind three. School 
two had a AA2: to 56% group composition range v/hich 
is still within the desired AO'^ to < ')% range. » 
Finally, priority A had a deviation :>f 295 whfth 
means the total busing distance for this situation 
is A095 miles. 

3. Job Factor Compensation 

This example is a summary oS J.O. Gun<iersbn*s 
doctoral dissertation (1975). It concerns the de^ 

>elopment of a model for determining jpb factor 
compensation for supervisory oersonn<?l under < 

collective bargaining. The basic idea is to dls-.. 
tribute wages to supervisory personnel where the 
dollar amounts desired exceed the dollar amounts ^ 
available. The situation models collective bar- 
gaining between the ^supervisory personnel* and the 
board of education of a school district . The 
supervisor's jobs' were assumed t.o be composed of 
twelve variables. The variables can be summariaed 
as follows t * 



Variable 
X, 



3 



Definition 

District advisory comittee 
participation 

Administration of -'.ifferen^ collec- 
tive bargaining contracts' 

Scope of work 

Budge t^'developnent involvement 

Supdrvisijn of nultiple programs 

Supervision t'icr supervisory 
staff 

Supervision of classified staff 

Responsibility for^^capitol equip- 
^ment 

Direct contact wit'.i "students 



11 



M2 



Highest degree obtained 



Supervision of staff under a fair 
dismissal lav 

y 

Years of supervisory experience 

Thus each supf^vlsor considered had .^i^^tajelve 
item *'job factor profile." TlUs iob factor pro- 
flle^determined 'the iimount of special compensation 
an individual would receive beyond his base/%alavy. 
Each variable was also scale^^.to reflect an inter-, 
nal hierarchy of importance within each factor. 
Each individual's salary was tlien the *um of his 
base stlarf plus the total dpllar value of each 
variable times the individual's aonropriate scale 
factor. The ?.o'tal bpdget fpr all cjperviSors' 
special compensation was the 'sum of the individual 
special compensation salai'ies. "his is an impor- 
tant* sum because linjits jVlaced on this total will 
affect the basic alignment betvreen each / 



387 



14 



4 



individual's, special coopensatlpn.** 

Six basic types -of const rai/its' were fornu- 
lated'CD reflect the relacionshipa becyeen the 
variable's. The first constraint was the total 
district resources , allowed for special compen- 
sation. The second type were the variable' 
value constraints used to force equality aoon^ 
the Variables unless the goal priorities affect- 
ed them otherwise. The third type was the nego- 
tiation constraint and it^was used to balance 
the total value of variables 1-6 against vari- 
ables 7-12. Fourth was the factor sum con- * 
straint which was used to provide overall model 
factor consistency. The fifth and sixth types'of 
constraints were used to ct5ntrol the scale hiarar- . 
chy widths and midpoints respectively. The total 
nuaber of constraints* (rows) was 23. Rows 1-13 
d^lt with individual job factors and rows IJ*p-^ 
dealt with the relations of job f actors, aixi'^r- 
sonnd types to each other. After internal scal- 
ing of each variable there were 3A9 totaL factor 
weights. , ^ , 

* Six priorities (goals) were established to 
reflect the overall view of how the job factor 
coKip^nsation laodel should work. Goal I was to use 
all of ^he YesourcGS allocated by the di<;trict for 
special compensation. The desire;^ level was %o 
Ifti^t un«lj?tsachievement of the total resources^ ' 
allocated.* Goal 2 was* to balance board initialed 
and supervisor initiated f^ctor^. Goal 3 wa? to 
balance the effect of the width and midpoints scale 
factors. Goal 4 Vas to maintain overall consis- 
tency of special compensation for the supervisors. 
Goal 5 was to maintain overall consistency and 
equality between each of' «he 12 variables. Goals ' 
2-5 wanted then to limit both over and under 
achievement. Finally goal 6 was to allow the^ 
total resources spent to exceed the limit desired 
in Goal 1. That is, the desire was to allow over- 
achievement within reason. TW.S was done to, allow 
the othei'' goals a greater chance of affecting the 
final solution. 

The initial district resource^ budget for 
special compensation was set a $58,3^5 <row 1) and 
each individual job factor (rows 2-l3) was .set at 
S167 ($58,395/3^;9) for each scale level. The solu- 
tion values of the variables then reflect the rela- 
tive importance of each variable in view of the 
constraints and goals used. 

THe results are as follows: 



Variable 



Value (in dollars-) 
225.10 

re 7. 00 

167.00 

i;>7.oa 

167.00 
167,00 
167.00 
205.40 







186.7a 






119.50 






205.40 




•^12 


186.20 



These results indicate that variables 1, 4, 8, 
9, 11 and 12,wdCe t^e most important sincq they 
exceeded the base value^f $167. Variables 2, 3, 
5 and 'h equal the base value an^ variable 10 was of 
least importance in this formulation. 

Four similar models, fomulated by changes in 
various pH.orities, weighting factory and goals, 

""tested as a part of the staJy. The result of 
model testing was a conclusion i'\at a goal piro- 
gramming model was developed which did demonstrate 
the capability to develop a job factor compensation 
plan in a public school* setting. The n)odel vl;as 
able to relate goal statements <S{ a prioritized and 
weighted nature to a series of mathematical reia- 
tionshi^^l^od prcKj^ce^ useful output for the 
decision-ma?<lng process. 

* - • Conclusion 

The adequacy of any single set,rf output is . 
dependent upon environmental and human coitslder- 
ations that are beyond the scope; of any model. 
vHowever, by using a tool such a? r.oal programming, 
a significant aid is provided Co tli^ decision- 
malcing pro<fess and the consecuences of a given set 
of goals can be evaluated ahead -of* time. 
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HUNTING BIG GAME— Wliy A BOW AND ARROW 



Dale Woods 

Northeast Missouri State University, Kirksville, Hissojiri 63501-^ 



AbSTRACt: I will show how one lany take a small con?>uter (one that doesn't even have, double precision), 
ui>t. Eatntmatical ingenuity and determine whether a large number — say over 1000 digits — has a small factor 
(lesb than 1000000). I also will indicate how one may usp a small computer in the search for large ami- 
cable pairs — over 1000 digits. Also, I will present ^ome currently unpublished amicable pairs as a part 
of the 'lecture. 



Most of you will* recoViize* the title of this 
paper lb similar to a previously published paper by 
Profe&Si^r D. N. Lehmer. I do not mean to itply that 
Uie big gaoki for which 1 am proposing to hunt is as 
largtr as that stalked i>y Professor Lehmer or that 
the speaker has^q<lal to or greater than his mental 
acumen in order to txap 'the elusive big game. How- 
ever, the speaker hopes that he will be able to in- 
spire some promising student to ^o after big game 
in the future. 

Obviously, the sjjeaker does not have access to 
Che hunting equipment currently being used by many 
other aaatnematicians (hunters), therefore, the sub- 
title "with a bow and arrow" seems to be appro- 
priate. 

i4 one goes on a hunting expedition, there are 
several things one tnust consider: 

1. What sort of big game should one attempt to 
catch? Our environment, number* theory , abounds 
with big game, unsolved problems, hence one must' 
decide whether it is the "neally big" animal you 
decide to try to catch or is one to be satisfied 
with a smaller trophy to hang in his -den for. a ' 
conversational piece. ' 

2. Is there the game available or what is the 
probability that one could find an animal that has 
not already been captured by a previous hunter? 
There may be that the particular game you desire 
Co capture is as extinct as a dinosaur or that all 
of this game is like the whooping crane whereby 
all are known and arc protected. 

3. What territory does the particular game in- 
habit? Would the animals vrobably be in a region 
you would be able to rc^ch with your method of 
conveyance (mental and physical endurance). Host 
any amateur mathematician could have performed 

the factorization feat of Profeasor Cole but not 
many of us would have given up the Sunday golf or 
the Sartur(fay night beer to devote his energies to 
Che division of integer^. 



4. Do you have the proper equipment to cap- 
ture the particular animal? An elephant could be' 
a prize if one used an elephant gun or perhaps a 
poisoned dart, buc one wouid be a bic bold Co go 
un an elephant hOnting safari with a muzzle load- 
ing gun. ^ ^ / > I ' 

5. After che previo*^ quescions have baen 
considered, one should chink through the following 
^question. How does one make , proper use of 
environment one finds himself? Where does one\use 
the computer at his command and where does one Use 
his mental abilities with a paper artd pencil Co 
minimize Che time spent and to maximize the output ^ 
in trophies collected? • ^ 

6. After a foraging expedition, does one come 
home with a prize? Could your "butterfly net" be 
empty or does one return home like the deer huntej 
with the prize atop his automobile for all the 
world to see? ^ 

Let us consider the above iii!^eater detail in 
the remaining timd that I have V4 discuss "one par- 
ticular safari with you. Perhap^ 1 should say bore 
a captive audience yith small talk. 

Of all the unsolved problems (uncaptured garfe) 
in mathematics one lOf^the most fascinating, to me, 
are the perfect nunbrfrs't? ts there an odd perfect 
number? Are there infinitely pany perfect numbers? 
Anyway, only twenty-four have been found (captured) J 
and at that only twelve since the conputer (^ost 
bow and arrow) era. Should one attenpt to locate 
an odd perfect number^ If one exists it must be 
greater than 10^^. From previous mathematical 
experience, it is indicated that if any odd per- 
feet nunfcer exists it will be a very large nunber 
and not to be found by direct computer search. 
Should one .attcnpt to find the twenty-fi fth -perfect 
number or attenpt to prove an in finite* nunber 
exists? Surely by use of a ^bmputer a rigorous 
proof of the number of perfect nuirbers is not 
probable and since the twenty-fourth perfect num- 
ber (2l9936)*(2l9937 contains some 12003 dibits 
it is very logical to "shy away from a search (safari) 
for this type of game. 
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A generalization of a pertect number is the 
amicable pair. Any perfect number being amicable 
tO' Itself. Are there infinitely many amicable 
pairs? .Is there an aml^cable pair of opposite 
parity? At present, both of these questions are 
still unanswered. We do know, however, that more 
than 1100 'pairs have been found and that if an 
odd pair exists then the product of the pairs 
must be divisible by at least twenty-one differ- 
ent primes. 

Protessor H. L» Rolf was the first to dis- 
cover, in 196 4, an amicable pair usin^ a computer. 
Prior to this time, some five hundred thirty-five 
pairs were known (in May 19A3— 390 pairs). All, 
of course, were of the same parity. Four pairs 
were published in 197A and at least three pliblished 
in 1975. »Since several others have locai^d (been 
abUi to capture) amicable pairs very recently it 
seemed to nc that this was the .type of game 1 
should attempt to capture. for a trophy. Since 
there are many d^^ff^reat varieties (classifica- 

, t,ions) of these beas.ts some decision had to be 

made as to ;)ust what type would be sought. I de- 
Aided to limit ray hunt to two very simple varie- 

^ties. 

All amicable numbers M, N must satisfy the re^ 
lation%S(M) = S(N) = M + N where, for the benefit 
of Donna, my secretary, $(K) is the sum of the 
divisors of tC. Using this reflation a systematic 
search is easily performed bn a small computer. 
The first such search prod^ioed some new pairs fcHit 
most of all, all pairs less Xhafi 10^ were now lo- 
cated. The systematic 'searcti h3s later been* e:^- 
tentied so, that it is believed that all pairs less 
than 10 have been tabulated^ If we consider the 
special cases, then there is*some -possibility of, . 
locating some >new pairs.' Firsts we restrict M« 
Epq and N « Er. Contrary to S^ioe previous writers, 
» I find that in studying this pro\lem it is, conven- 
ient to write solutions in the lotm 

Q « C/D -1, C « E*(P -K 1) ,\ 
D - E*P»-(P + 1)*(S(B)-E) , ^ 
R « (? + 1) * (Q + 1^ -I. 

then assign prime. values for P and teit the pri- 

2^^ an{i\ ' 



2^-M*jrVri)'-l we haCe 9 - 2f^^(2^ +* 1) -1 » 
R . t^^^^*(2M + 1)2-1. 'For different 



of Q and R. Now if E « 



mality 
P - 
and 

values of M and N this form has been studied for 
many y'ears. Several ioteresting theqreos re- 
garding the primality of the F^,Q, and R for dif- 
lEercnt values pf M apd^'iaay b^ proven. I have 
found several interes'U^g relationships that as 
yet have not been published so we^wlll not dis- 
cuss ^these at this' time. « • 

If M * I i^ the abo)e we have the formula -de- 
rived by a 9th century Arabian mathematician Kor- 
*rah. *To ray knowledge he did not locate. an aml-*^ 
, cable pair by use of this formula. It, seems th'at 
Fermat foimd the first jafr In 1636 from this for- 
mula. Tlie qost recdnt was found by Professor 
te Riele in 197 4 using N - AO, and M - 11. This 
latest pair have £orty digits each. ^ 

I first calculated the pos'sible pairs for 
M - 1» N < 200 not because I wouid hope' to find 
any new pairs since it has been wcll-Jviown for 



quite sometime that none existed. I did 'carry but 
the computations up to N » 400. According to ray 
calculations, there are no pairs in this range. I 
then calculated all values of M *and N up po M ^ 150 
I must say that 1 did get the pair for A = 40 and 
M ■ 11 but unfortjun'ately , Professor te Riele had 
found this pair some three or four months before 
n(/ work was completed. In this instance, my 
"butterfly net" was empty." I do plan to publish 
the results of this work in the* near future. ^ 

N 

My next step was to let E » 2 * S. . Only five 
of this typb had been found, however, I suspected 
there were more around. That is, game surely must 
be found of this type and in the range that the 
small computer couUi reach. Tlie basic formula is 
Q « (E*S*(P + 1))/((P + 1)*(S + 1 - 2 * E)-E*S)-1. 
One now assig;ns prime values for S and P then com- 
putes Q and R. I restricted N to less thai sixteen 
since on a small computer (Honeywell 1640) I founjj 
that I cannot work effectively with numbers greater 
than 8300000. A program was written with these 
restrictions. After using 28 seconds of computer 
time, I found only one amicable pair, the one 
known to Pythagoras. 0b\ziously^ the program needed 
to be altered. The next program was written to 
take advantage of the fact that the denominator 
should be a factor of E*<P+1) . We then let 
Q»((E*(P+1)/((P+1)*(S+1-2*E)-E*S) /S)-l. I further 
restricted S«2*E-1+2M for M - 1 to 2*N. This pro- 
gram produced five amicable pairs of which two were 
known. Those unpublished are 

23039 * 811259 
ltol430399 

210*2783 * 1077413951 
2265568819642367. 



2 * 



2 * 



4507 * 



4099 * 



an d 



'have^ieen forwarded to a journal for p'ubli- 
an. The oasic program has been altered so 

-l+L,for i*2 to 2*E. With this change, nine 
s'were obtained of which five -hai pre\dously 
^'published. ' The other two unpublished nuiii)ers 



2 * 



23423 * 



23030 *»292872959 
6747972998399 



and 



0^^* ccmi * A359101 * 139491263 
2 * 66047 * 668056647884927. 



tt qne has the bi^ gun (a con5)uter with inte- 
ger double precision ox a multiprecise arithmetic 
program) one would f>e encouraged, to go dfter the. 
bigger game, ..But without this equipment one must 
4io some reflection before attempting a safari. 
One could, of course, do as one mathematician did 
in the past. Have his students (slaves if you 
dare) perform the tasks on a hand calculator which 
has a considerable number of digi^ precision. * 
Anyway, one with no slaves and with only a'bow and 
arrow must make other adjustments. f 

If one does not have the multip^recise arith- 
laetic available one "can create one by using bases, 
other than 10, After. doing this (poisoning the 
arrow) one is ready to proceed. The basic idea is 
to simply use blocks of digits. For exatn)le, if 
n-853476 then we use N(2)-853 and N(l)-47^. One 
now can easily perform the four basic afibhmetic 
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operacio;\s wrcli numbers up Co say lOOO digits quite 
easily. To add or subtract, simply calculate 
C(L)«A(L)+ii(L) then normalize with base 1000. To 
multiply calculate C (K)-C (K)+a(I) *B( J) where 
K«I+J^1 and I, J range over the nui^ers in the mul- 
tiplier and multiplicar^d then noruiftlize. To di- 
vide is just a bit* more difficult if the divisor 
is greater than 1000, tiowever, if the divisor is 
less tnan 1000 all one needs to* do i-S to calculate 
ii(L)»LirjA(L) /D) then normalize, 'lo generalize 
tne, divisor to numbers greater than 1000 but less 
•than 1000000 we. calculate i>i»A(L)*lE>l-A*(L-l) so 
giL>«li>iT(i^/U) then follow the fundamental division 
aigoritiim. Witn numbers ^gretfter than 10^ the trial 
quotient gives a bit more trouble; A further com- 
meot* 15} that we are of course limited to the size 
of the computers memory for subscripting. 

After calculations o^f the'Q and R satisfying 
the equatiort *i»r amicable pairs the next step is to 
determine if t\cse computed values are prime or 
composite. We need then an efficient method of 
factorization for larg^ numbers. To locate factors 
less than 1000 ofte of course just uses the division 
algorithm repeatedly. I wiM not dwell on the fac- 
torization prodtss since this is very well covered 
in the recent work of Professors Selferage and Guy. 
I should, however, remarJ^ tnat Professor Shanks**has 
developed a process^ to factor twenty-digit^ 
on a hand he\d calculator. ^ Since the Q an3 RT f or 
this second proble!|^are quite small the method of 
SnankS'is quite satisfactory. 

As previously noted our "butterfly net" was not 
empty. Further ejcpeditionik for hunting big game 
pcgoably will iOe quite productive. I suggest that 
you spend some Sunday afternoon (not as many Sun- 
days^as Cole) on a hunt. . In my opinion, there are 
many trophies available. In fact, so many that 
they do^not make* a particularly good tropliy (not 
many are publlslTCd except in the WOTlCES) any more.- 
fne major deterrant seems to be in 'the *area of 
factorization. One might say-to determine if that 
which. we have captured is not in season. 

yay 1 suggest thAt future computer oriented 
mathematicians continue working on more efficient 
nwthods of factorijtation . When this i$ accomplished 
I ^eel certain otUer big game will be brought in 
as trophies for all to view. This is especially 
true if we consider the case where* E ■ 2^. In my 
opinion, many ^ibre pairs may be found by con^ 
sidering ' 
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It has beec^ said that tlie factorization of numbeVs 
up .to lO^y vjill not be accomplished in this cen- 
tury. Pe/th*aps this prediction is like one of th^ 
past whereby it was indicated t^hat it would be 
quite sometime before Aunfiers of 40 digits were 
factoroU. Surely the mathematicians of the next 
20 years will be at^le to do -much more in this 
area fhan the mathemQticians have accoflplisht>d in • 
thescy past 20 years. One mathematician has de- 
veloped an interesting method of factorization by 
assuming Riemann's hypothesis. 
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DATA PROCESSING AND SCHOOL MEDIA CENTERS 
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Don D. Shirley 
Supervisor of Library Services ^ 
Shawnee Mission Public Schools, Shawnee Mission, -Kansas '66204 



ABSTRACT: Our district of 40, 000 students is using data processing in the school media center pro- 
grams for central library processing, d^ta retrieval and acq^^isitions*: The major function of the com- 
puter is cei\tral lij^rary processing. This is on-line thro.ugh trfe use of four 3767 terminals, utijizmg 
IBM.'s FASTER gener.aRzed software package and the district's IBM 360/40 computer. The retrieval as- 
pects of the system include bibliographic data, Inventory data and printing of extra cards and labels. Se- 
lected tapes are used.to generate microfiche union catalogs for our high s.choQl collections of 80,K)00 
items. Our idata bank consists of 152, 000 unique book and multimedia titles. The cost of the program 
has reduced from $4.30 (J969-70) to $2. 18 (1974-75). ' . . 

towards freeing .the libj-arians'from clerical- 
type functions and to increase the availability of 
resources for both student^ and teachers. The , 
lib*r»rian^s time c^n then be rechanneled towards * 
worlcing more closely with students and teachers. 
In addition, we draw statistical information which 
. assists in guiding sound management prp^ced,ures. 



I. Introduction 

Shawnee Mission (Kansas) Public Schools Is 
located in the suburban portion of the Greater 
K'ansas City. Metropolitan area. The district en- *; 
r9llment is around 40,000 students; kindergarten 
through twelfth grade. There are 49 elementary 
schot>ls, ten junior high schools and*five senidr 
high school^. The district was unified in 1969 , 
from 12 elementary school districts of various 'Pv 
sizes and one high schpol district^which also in-. ^ 
ciu<5ed junlAr highs. The highesl enrollment, al- 
most 46,000 students, was duriiig 1970-71 schooW 
year with a steady decline of severSil hundred stu- 
dents a year sinc« then* During the 1-974-75 
school year, two schools were closed and it is 
anticipated ovei: the next five years several more 
may be closed. 

Each of the district's 64 schools have a 
school library media center. All but 18 school 
media centers/'have a full-time lit>rarian. The 
collection si7.es in the elementary schools range 
from 4000 to 16,000 books and 500 to 3500 multi- 
media jaudiovisual) items: junior highs averag!<» 
12,00^ books an^ 2500 AV itemsj and the senior^, 
highs average 20,000 books and 3500 AV items. 

.The district has a Department of Library . 
Seiv'^ices, which coordinates all of the library'^ » 
rela'ted'activities. This department has three 
\full-time administrators, US full-time certified 
^ cataloger, one classified cataloger,- and 20 • 
clerks. They are responsible for a district 
Film Library, Central Library Processing, 
Library Material Acquisition^, library related 
federal funds. Professional Library, preview 
j^nd examination center, etc.; The Central 
Library Processing Center r'cQclves, processes 
and ships to the media centers all of the library 
material purchased. 

The main purpose of the policy of the 
Shawnee Mission Public Schools irt regard to 
automated librai^ processing Is directed 
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Immediacy is a key in ^Jroviding quality li- 
brary services to students arid teachers. They 
require assistance in locating library resources 
when the need is felt, not two weeks later. Auto- 
mated library processing provides the splution 
to this^roBlem by releasing the librarian f rom . . 
clerical duties $o she can help the students now 
and by making available data^oti resources In the 
entire district, thus providing maximum service 
for Che student . — 

The day of -the students or teachers using 
Qnly their, own school'^ library is quickly fading.* 
Automated processing allows tlje students to gain 
ac'cess to ^11 64 libraries in the district. 

The district utilizes Data Processing in ' 
*many of their media functions. ^The primary use 
of the computer by the media centers is for the 
Central 'Library Processing' I.CLP) activities. 
Data Processing is also used^ fdr acquisitions.^ 
information retrieval, magaJN.ne ordering, over- 
due not^es and business fupchons. This paper 
is designed *to delineate aU'of'these uses. 
\ ' . . - , 

Technical aspects of <;hc. district' s computer 
center and operation is giv^'ett f.irst to provide the 
reader vJith a basis- to' refei- back to as the vari- 
ous data processing uies are explained. 

Technical Aspects ySoftrVafe 

The software framfiWoh> for the on-line cata- 
loging is FASTER ^Filing an^i.Source Data E>itry 
Techniques for Easier ^©tyieval). This was first 
developed In Ala?neda, CidiTbrnia, for law en- 
forcement. The designer^ of the school district's 
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program observed this system in operation at 
the Kansas City (Missouri) Polipe Department 
and felt it would satisfy their needs. FASTER 
was selected as the basis for the on-line cata- 
log system in January, 1970. i 

The main language for the other data pro- 
cessing activities related to the me^ia centers 
in the district is COBOL. Batch work related 
to centralized library processing such as the 
printing of cards and labels utilises COBOL 
also. s< 

Hardware 



TABLE I (continued) 
Copy 



The district' s computer cent,er consists of: 

1 IBM "360/40 computer with 'DOS 384K^ 
8 discs 



4 2401 tape drive 
r 2701 line control 

1 2540 Card Reader/Card P-dnch Unit 

2 271 1 Modens 

I 3*^47 Data convertor 

1 optical mark scanner 

2 3742-dual data stations 

1 1403 N 1 Prijiter (with upper-lower case 
print train) 

At the Central Library Processing Center(CLR 
are four 3767 terminals. 

The copiputer center system operates in 
three partitions. Partition Fl houses Power 
Spooling Support. FASTER is housed in parti-, 
tion F2. Partition BG is used for batch j.obs 
(COBOL). * . 

Files 

There are two basic files for storing the 
Cataloging data. The first is the Title file con- * 
sistirifi of full bibliogra'phic data for each unique 
title in the school distrio^«. Table 1 displays the 
information in the Title /file. • Each unique Title 
record is 562 characters long. 

TABLE 1 - Main Fields Input, by Operators 



Record Field 

Title Form code 2 

Publication date 4 

Copy^i^ht date 4 

Author 35 
Title ■ 50 
Annotation 105 

P.ublisher 30 

Edition 3 

Price 5 

Dewey tiumber 8 

Cutter number ^ 10 

Grade level 4 

Collation 40 

Se;:le8 35 

Canguage code 3 

LC card number 14 

Subject heading I 24 

Sub)ect heading 2 46 

Subject heading 3 60 

Added entry • * 30 

Copy Title number * ' 7 

Number c/f .copies 4 

Building code S 

Funding code ' Z 



Length Comments 



Distinguishes 
/phys ical format 



May be continued _ 
?in Annotation ' 



r 



Use MARC lang. 

/a*bbrev. codes 
Use Sears headings 
Use Sears headings 
U.^e Sears headings 
For name or: tifle 



If other than gen. 



Volume^umber 3 For vol. /other seq. » 
Print instructions 16 Kept bnly until 

^ ^ _yiabels^>: cards print. 



During q^n-line operations, 8 a.m. to 5 p.m.", 
Monday-FrCday, a modified Title file is being 
used. This modified 'Title file consists of a 
shortened title (first 50 characters^, the author's^ 
name, physical forpiat'code and the unique title ]/ 
^number. The modified file was developed so 
that additional disc packs could be released dur- 
ing tde day f^r other uses. This Title file is' 
stored on two disc 'packs and has the capability 
of handling 190, (JOOf title records and 9J0, 000 
copy records. Wh^fen printing or ret|^ieval func- 
tions are in operation the full Title and Copy 
files are brought u^ and run on six-disk -packs. 

The second ba^ic disc Jile record is the 
Copy filfe. The Co^y file contains a 56 character ' 
record for each copy of a title, and comprises 
fixed-length fields for building number, special 
funding, yolume information and circulation con- 
trol. Co^y and Title records aile linked through 



A thir^ file is the ISAM (indexed and sequen- 
tial) title index, comprised of recyjrds with a 
.phonetic ctjae key for each^title r*cqrd. This 
file is calledyup by a terminal trahWction <jon- 
taining title, the incoming phonetic code for t^e 
title Is matched with any equal ones on the index. 

In the cataloging files each unique title is 
assigned a unique title number. This title an5 
copy number consists of six /placed to the left of 
the 'decimal and four places to th^ right. The 
title number is represented by the first 6 digits. 

Example: ' ^ ' / 

Book Cat in the Hat 

Filmstrip Cat in the Hat 



013469 (Title No. ) 
002387 (Titles No.*) 



As schools acfd this book or filmstrip to their 
medja collection the numbers to the right of-the 
dechnal are assigned automatically by the com-^ 
puten. Example: 

Books % 

School A 



Apache- 
Oak Park . 
Pawnee 
Apache' 



Cai Ihlhcr Hat 



FilmYlrip^ 

School 

Traitwood- 



/ ^lint 



Cat in the 


Hat 


Cat in the 


Hat 


Cat In the 


Hat 


Cat in the 


Hat 


Cat in the Hat 


Cat in the 


Hat 



01346^. 0001 
013469. 0002 
01346^9. 0003 
013469. 0004 (2rti oopy) 



002387. 0001 
*002387. 0002 
002387-0003 



)\ complete and ^very detailed description o ' the 
technical aspects of our on-^lino cataloging sys- 
tem can be found in the followlrrg article: 

Miller, *Ellen Wasby and B-J. H^dgps, "Shkwnee 
Mission's Catalogl^ig System^', Journal of jLlbr&ryr 
Automation , 4:1, March, 1971, pp 13-26. 
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In addition to the catalogiHg system files 
there ari separate files for the acquisition- syb- 
tern, the overdue system,. the magazine ordering \. 
system, and the business system. The iirfonna- 
t'lon retri<?val system utilizes the cataloging sys- 
tems data bank. ' 

II. Central Librar>' Processing (CLP) \ ^ 

History 

The yeai^ before uni/jcation, 1968, the IJi^h 
School District started an automated system for 
cataloging librar>' material. This \sas batch 
system using an IBM/1401 computer. At the 
time of unification, 1969, only the high school? 
and two other elem^tary school districts, of thfc 
13 district, had centralized library processing. ^ 

W.ith unification the new distFict committed 
itself to centralized library processing for the 
65 schools. To facilitate the handling of the an- 
ticipated mass of library material to be proces- 
sed, it was also* decided, to stay with an automated 
system. The^high school 'district's batch system 
presented a number of problems, which would 
have .to l;e overcome*, to establish an effective 
and* efficient operation. Some of tl)e problems 
were : 

a> Limitations of the 1401 batch system 
to handle 100,000 items a year. 

b. Only upper case letters on catalog tards. 

c. ' FLxed length fields whi<;h resulted in ab- 

breviations that could be di/ficuU ^or 
'the elementar)' age children. 

d. , nigh incidence of error,. Need for more 

accurate data on cards and Eabels, 

e. Conpilations with paper handling; color 
code grid- forms for duplicate items, 
massive 'shelf list, etc. 

^ After considerations of these problems by 
the'disj.rict's personnel in both library and. data 
processing it was decided to explore the possi- 
bility of some type of on-line systeJn. Tlie on- 
line system became a reality in mid->larch 1970 
on a pilot basis. By /\ugust, 1970, the total pro- 
cessing was converted to the on-line system. The 
on-linei system solved most of the problems that 
existed in the batch system. A major break- 
through was scored with the capability of upper 
and lower case letters on the catalog cards,' thus 
making reading of these cards more natural to 
' the children using these cards in the school 
' media centers. 
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.The syst^ has ^rown from one 2740 t-enni- 
nal to four 3767 terminals this yeaf.^^ At one 
tijne in 1971 the Central Library Processing ' 
Center (CLP) was using ^ight 2740 t^srininals. 
The data^bank has gone from zero 'to ftO,0'00 
pities and over 500,000 copies. . All library 
material purchased in the district for any school 
media center since August, .1970, h^s been pro- 
gressed by the on-line systcin ot the CIJ^. 

Present application * ^ 

CLP receives, catalog*; and processes all 
librar)' material purchased^ for chfe" school media ' 
centers. The items are checkedVia the termi- 
nals to determine whether, we ^'^Vo other copies 
in the district. If it ,^s;a dii{?iication, then 
the data bank is- up-datc^1 on-line tp add this 
item to the bank. 



If the iten) IS not a duplicntion then it i^ sent 
to professional catalogers to be cataloged. After 
original catalogingt^ is returned to the terminal 
operator to inpuj^he item's bibliographic data in 
the computei^. A conplete set of catalog cards 
IS printed by batfh operation in the computer 
center during the evening and night shifts. Also 
labels* for the book pocket, check-out cards, and 
book spine are printed by batch. These cards 
and lablls are therj sent to CLP for the final 
processing. 

The entire opera tiort tak«s any^vliere between 
five days to five weeks to complete depending 
on the time of year and volume of new' material 
in CLP. The average amount of elapse time 
between receiving the material in the center 
and shipping' it out to the building is between 
three and four weeks. 

Data Bank 

The data bank developed by adding this ma- 
terial to tjir*med4a centers is the major involve- 
ment with the district?' s Data Processing Center. 
This cataloging data bank is gr<S^ing at the 
mont^iiy rate of approximately 1200 new titles 
and 7000 copies. Table 2 displays the' data banfe 
size as of february, 1976, 

TABLE 2 - CATALOGING DATA BANK ^ 



* Physical Format 

BOOK ' 
SOUNDFILM 
DISC<J^ 
TAPE' 

.j::assette 

SUPER 8- 
FILM 16 

SLIDEH, 
FILMSTftiP 
MDDEL^^ » 
MAP 

GLXRT I 
FUSH CD 
r,LOBE 

'kit 

GAME 

VIEi\MSTR 
PRim-'AP 
PRim- SP . 
REAt>IA 
SD SLIDE , 
^D tOpP . 
MI SLfDE 
^ m LObP 

"ftEPRI^nr 

PAPER BK 
PAPER BD 

Totals 



Copi6^ 

388801 
19(W5 
12030 
2035 
. 9SU 
1W58 
^965 
4612 
. 1994 
' 3§173 
305 
184 
290 
. 84 
25 
994 
770 
382 
^875 
5525 
34 
708 
i27 
57 
65 
578 
15141 
1320 



Titles 

106611 
6351 
5293 
1379 
2847 
3770 
1682 
1854 
831 
10514 
222 
124 
' 205 
30/ 

34i' 
231 
2080 
1455 
20 
233 ' 
8 
48 

> 

* 481 
5167 
778 



The steady growth of this data bank can be seen 
in the four-year growth- patterns of Table 3. 



Form_at 


Mkr(h 

1 y\ 0''5 


1972 




1076 
Tltlr* 


Added »n 


4 vr*. 

r\l\cn _ 

55. 056 


AOS, 262 




752.207 


Multimerti* 


-17. 2H0 


10. (Ay 


114, 327 


40 /.7^ 


(tl.OAT^ 


20.0f>6 


TOTALS 


?00. 3)S 


78, 113 






1I'J.2S4 


7A, 116 
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A detailed anaiyh 
Table A, im)^aCes the 
is aboat^fsto 1, with 
ratio in^»oks 3.6 to 
This table also shows 
and 73% of the title f 
a .wide variety of mult 
books to multimedia is 
our total media center 



is of this d^ta bank, 
overall duplication rate 
the largest duplication 
1 and filmstrips 3. A to 1. 
that 77% of*the copy file 
ile is books and the others 
iujedia. The ratio of ^ 
fairly representative of 
collect ions. 



TABLE 4 - A^At^Sl5'0^ CATALOOrNC PATA DANK 





No o( 




% of all 


'A c( ill 


'"bwp r»l.o 


1 


Form •» 


















Hi. s>s 








c . 


MuItimwiliA 




IU.327 


40. 




27% 






rorm-st 













nc»».rlj>»ioti 
if fc»rnn»t 












3 4 


>v>UlwJ .|rt4l 


I«mn\ rilint 


1, vts 


I,65Z 


.4% 


• in 


K2 






2 3.',?«. 






6% 


2,5 


fJi»c. /*;><■, 


P milt Kiln.i 


11. 




2% 


3% 




ftPSul^i 


Print* 


<», 10 J 




2*; 
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We do not presently have all of the library 
material on the computer. In May, 1975, we 
compared our data bank totals yith our inven- 
tory figures. .We^ had 651,366 books in all of 
our media, centers and about 56^ of these, or 
362,141, were in the data bank. In the area of 
multimedia, the school*i:lnventoried 126,490 in 
May and about 83% of dhese (105,000) wer'e on 
the computer.^ In h May, 1972, compai^ison of 
these same figures, we had only 26% of the dis- 
trict's books on the computer and about 45% of 
the multim.edia items in the data bank. 

A long range projection of our data bank > 
indicates we anticipate that the unique title- 
'count will level out at about 230,000 and the 
copy file to be almost 900,000. tf all factors 
remain the same we are expecting to reach 
this level by May, 1981. This projection is 
based on several factors: 

a. Four terminals inputting 85,000 *' 
items a year; 

b. Keeping only 4 terminals; 

c. Status quo of library budgets; 

d. Cost of library material increase " ^ 
about 10% annually; 

e« Annual deletion rate <or the schools 
of about 3.5% of their collection size 
(lost, damaged, discarded material). 

Retrospective CatTaloRing 

As previously pentioned, everything pur- 
chased for the media centers since August, ^ 
1970, has been added to the data bant. Most 
media centers were established before this'' 
time and so plans hat! to b6 made to add these 
older items to the data bank. The teihn used 
for adding- these pre-August, 1970, items to 
the data bank is. called "retrosjpective citaXog- 
ing." 

At least four different methods of retro- 
spective cataloging have bqan trle^: 

a. The shelf list card for each item is 
sent to CLP and this data is entered. 

b. The actual physical items n6t on the 
'* *' comp^jter are sent to CLP. 



d. 



Cards of a selected section of the data 
bank are matched with the media 
center's shelf list. The cards that 
match are sent to CLP and the data 
entered. ^ - ^ 

A terminal is moved to a media'' center 
and the material entered at the building 
site. ' f 



Of these methods, the second one of having 
the actual material sent to CLP has been tiie^ 
most successful. This allows the t'erminal op-r 
erator to actually compare the data on the ma- 
terial with the data bank information. It ah^o\. 
allows CLP to finish processing the Items coa- x 
pletely so that when it is sent back to thq[ school 
it is ready to be put on the shelf. THe other 
methods require the librarian or building level ^ 
clerk to attach the labels sent from CLP* after 
the item has been put on computer. 

* The processing 'of^ew material takes pri- 
ority over retrosprfCciveV^taloging^ in CLP. 

Two techniques of retrospective cataloging 
have been performed in the district, The first 
technique was to select a sinigle school and com- 
plete retrospective cataloging on the entire 
school. During the 1974-75 school year a differ- 
ent approach was used. The district's Art De- 
partment wanted to know/ll the art material in 
all the media centers. All of this* da*ta vas not 
in the data bank. The decision was* made to do 
^retrospection across the district, in the Dewey^ 
/Decimal classification ( 700-769)' ^ pertaining to 
'Art. This^was dcrne and now we have all the art 
material in all 64 schools on the computer.. 

This year it was decided to continue with 
this approach. , We are presently involved in 
entering the rest af the 70O's to the^data bank. 
It" is anticipated that by^ the end of this year, we 
will have all material, book and multimedia*^ in 
the Devey classification 7^0-799.99 fttm all 
th& schools in the d^ta.bank; Ev'en while we are 
doing this, however^ *we are^^till at work on re- 
trospective cataloging, in beyer^l lnd{vidual 
schools. ^ 

After a^school .adds tts\material't'o* the 'com- 
puter, it normally ifeceives a new set' of catalog 
cards for this mat'erial. An Alternate method 
most often used by . buitdinge when theJ^r entire 
collection is added. Is to request a new^ card 
. catalog. 
• generate a. 
betized. 



'* ^ » — " — » — r- - c 

The computer, has the cHpabilit^ to 

3. complete c^jTd catalog alreadv^alpha- 



o 



Presently .in the district we l>aye all the 
material frqnf five high schools, three junior 
highs and five eCementary schools completely 
♦in the data bank. All of the schools' 700-769 
Dewey^'s are in. There are three junior highs 
' And one elementary school working rapidly to»- \ 
^wards being on the computer. It is anticipated 
that by May, 1981, all of the media collections 
for all of the sdlVools will have .completed retro- 
spective cataloging. 

; • Probleros ^ 
^ The system In general does not seem to 



/ 



have any major drawbacks. The problems 
related to it arc more similar to problems of 
Central Library Processing operations in 
general than related to Data Processing. * The 
problems related to fixed length fields seem 
to be the main programming problem. Catalog 
card format bothers some* librarians ; particu- 
larly the lack of t^e ability to distinguish ^be- 
tween a title and ^subject heading at the top of 
a card. Other problems are; 

a. Limitation of having only three 
subject headings per entry. 
Keeping price of material on 
catalog cards up-dated. 

c. Keeping data bank current. If a 
library deleters ah item from its 
collection, CLP must be notified so 
they can ^clete from data bank. 

d. If a typogr'aphical error gets into the 
computer it can repeat itself over ^nd' 
over until it is caught. 

e. ,If an item's classification number Is 

changed for one school, then all schools 

* t • , having thjt item must change their spine 

labels^Dook pockety, check-out labels 
and catalog cards. ^ 

The district has an active Cataloging Com- 
mittee^ft^de up of librarians from all grade 
levels, CLP catalogers and supervisor, and a 
Dept. of Library Services administratol^ This 
committee must approve any changes in Dewey 
classification and other major decisions affect- 
ing cataloging and the (lata bank^, 

Cost Analysis 
' -To determine the actual cost processing 
is very difficult. Most datar affecting this cost 
analysis can be collected, including cost of data 
processing and central library processing per- 
. sonnM,. terminal and line rentals; and an esti- 
mate^ of the pro-rated oomputer cost, and this 
data can, be analyzed fairly completely. The 
items, ^hich are mote difficult to measure, if 

* th^ dbcipicm -Ts made to even Include them in a 
cosi analyses, are the building maintenance 
costs, of -tpe facility housing central library _ 
prices stng (iticluding water, lights, gas, cus- 
tbdiafihel,^*, and* necessary repairs): the costof 

, developing. the baiginal coiAputer program (over 
how*rnan>rfears*^o you pro-rate this expense; 

, were 'accQrato* records kept In the beginning to 
show everyone' s^tlme'involved "in program de- , 

Velopme.nt, etc. ): should only the cost of pro- 

• ces^ng new ltetyis**be considered or should re- 
trospective /:a^al6g{ng be Included; should minor 
programmijtig (Changes bc*cpfnputed in the cost 
' analysis at^d for. how many years should they be 
tfpread.out; the cost of equipment (typewriters, 
embossing rpachine,- «tc. ); the cost of the busl- 

"ness offltse's fime relMed to central library 
processing: and, 8a*\Hng of librarians', clerks', 
and admlnl«t/ators* time when the data ba<ik 
is tisefi- tdW-rtish necessary ^ata. 

The^qudstlon'also arises' of whether to * 
"figure a cost a-nalysls on the full capacity of 
the system, 100, 000 items annually, or on the 
actual output within a given pc.rlod of time 
(summer months, semUannu^lLy, annually, or 
pver several years). There may be extenuat- 



ing circumstances which prevent full utilization 
of the capacity of the processing system, such 
as power 'failures, computer down-time, person- 

' nel problems, telephone line problems, loss of 
data due to computer problems, changing of pro- 
grams, personnel turnover,' training time for 
new personnel, duplication rate and type of ma- 
terial input (media items ^take longer to input 
thanbopi^s), etc. So the question arises, should 
the cast analysis be based on actual output or on 

\ what thjC system was designed to produce^ ' 

' k 

Analyses of estimated cpsts that have been 
computed on the Shawnee Mission Public Schools' 
"systeiii over the last several years haye been 
both on a full system' capacity situation and actial 
output. This includes data processing and cen- 
tral library processing" salaries and supplies, 
and an estimate of computer costs', terminal ex- 
penses, and other data processing expenses. 
During the 1969-70 school year, the system was 
on a batch system using keypunch to enter data: 
the estimated cost of processing a library ^^^'JL^ 
was $4. 30. • 

Table 5 displays the cost analysis since 
I969r70 and a projection of the current year, 
1975-76. Records were not accurately kept f^i; 
the 1970-71 or 1973-74 school years, so no cost 
analysis is given for thes^ years. This table 
shows a constant decline in processing cost until 
this year. 



co»t per actual , 
Item proQcsocd 


69-70 
$4.2^ 


71-72 
$2.42 


Y2-73 
4 2.56 


74-75 
42. 1^ 


Pfoj. 
75-76 

$-2.44 


cost por^jCoO. 000 
Items pr<^c5ftcJ 




$2.39 


$ 2.34 


$1,94 


$2.20 


Vumber <}£ Uemi 
pri)CAsl«d ( * 


22, 000 


98,788 


107 


89, 188 


90, 000 


system ln« 

oper^ion ^ , 


Hatch • 


on- lino 


on-line 


on-line 


on --line 



There are ijevjsral factors which influenced 
the cost increase for this year,. The major ^ 
reasons for the increase' a're an adjustment in 
the salary of the CLP 'supervisory position, and 
a greater pro-rated amount of the computer 
charged to library processing.* Last year the 
library system was run during the same time 
tHat a Computer Assisted Instruction^ (CAl) sys- 
tem for students and teachers was In 6peration. 
llie prorrated cost for each of these function's 
tended to balance the cost of the computer opera- 
tion touring this period of the day\ This year the . 
CAl activities have.been dropped ^o"^ 
trlct*s contputer/ so the library pfocessihg sys- 
tem must assume a larger portion dfthe cbm- 
putfl^r cost. This year's on-line comWter ex- 
pense increased by 56% over the I974v5 School 
year. • 

The Increased cost for processing would 
have been even higher If lt had not been for sev- 
eral changes.' This year four 3767 terminals 
are in operation as compared to five 2732 termi- 
nals in 1974-75. In addition to saving on the cost 
of terminal rental we were aljio able to reduce 



one t^errrttnal operator's position. This person 
is now being use<ypart time in the Film Libr,ary. 
Due to the high duplication rate and the size of 
the data bank we*were also about to reduce the 
amount of professional cataloger's time. 

Cost analyS'e^ by other districts having » 
«f^tral library processing range from $.60^to» 
$14.80 per library item. A study done by the , 
library consultant from Bellevue, Washington, 
in l968-b9 sho\yfd that to catalog and process 
one book under ideal conditions would cost most^ 
districts at least $2. 32, as reported In anAmer- 
ica;i Library Association bulletin. As you probt 
ably suspect, with the v^ry wide range in costs 
($. bO to $14.80), ^he people computing the' cost 
analysis are not talking about the same thitig 
when they think about "central library proces- 
sing*. " This same kind of Communication gap 
exists^when commercial library processing is • 
considered to be the same as Shawnee Mission ^ 
Public Schools' central. library processing\ 

lit. Information Retrie^ral 

The present retrieval system ts handled al-^ 
most completely through batch operation in the 
computer center. The data bank -used isf the 
same as tViat developed from the processed items. 
Even though this is a large data base, 150, 000 
unique titles an4 over 500> 000^ copies,, it does not 
represent the total collections of the district's, 
media centers. As more schools or district-* 
wide sections of the collection are added to the 
data bank, the retrieval utilization will increase; 
Some of the value of the retrieval system is 
represet)ted in the following list of capabilities 
and services. 

a. Provides bibliographies for -specific 
teaching units or selected readings. 

b. Provides bibliographies for use with 
(jurriculxim development task forces or individ- 
ual teachers. ^ 

c. Provides bibliographies on new ma- 
terial to the district in any specific time se^. 
quence. 

d. Establishes if a district has a specific 
item and its location in specific media centers. 

e. Pr'bvides data that assists teachers 
ai>d librarians in selection of material for their 
media centers. 

f. . Provides bibliographies of material in 
specific physical formats (cassettes, films, 
games, prints, etc.) which can be used to de- 
velop interest centers, provide students with a 
variety of media experiences and allow students 

^ access to material best tailored to their learn- , 
Ing style. ' 

g. Provides jStatikjical data. 

h. Reduces laper handling thj^ough 'pro- 
duction of additionSil catalog cards and labels. 

The information from the data bank can be 
retrieved by a number of ways. To date the 
most common retrieval requests have been. by 
Dewey Classification or subject headings, or 
for a single building location. Over the past 
few years programs have been written as 
needed for special requests. 

\^Vth Individual bibliographies the data can 



be retrieved in any number of combinations re- 
lated to special form codes, building locations, 
Dewey classifications and Subject Headings. 
There, are about 3-5 such requests a week. The^e 
retrieval requests are handled by completing a 
preprinted grid sheet, which is keypunched. The 
comp,ater then supplies the appropriate print-Qit, 
q^sually withir 48 hours of the request. 

Another retrieval technique which will prob- 
ably expand greatly over the next fe\y years is 
the request for single items. This is done on- 
line at our processing center. Usually ojice a 
day they are called by a librarian to determine 
if the district has a specific item and what metia 
center has' it. At th\B tirhe this on-line retrieval* 
can only be done ifa^specific author and title are 
kt)own. ^ ^ 

Special requests, usually resulting' in large 
print-outs, are handled on an individual basis 
with direct communications between the Depart- 
ment of Library Services a*nd the Data Proces- 
sing Dept. Due to the amount of computer and 
print tvn?e required for these requests, they are 
planned for on a scheduled basis. 

All of these retrieval requests are designed 
to provide teachers, students, librarians, and' 
administrators with data as to the availability of 
resources and to save time in papef* Handling. 
The data processing cost for these requests for 
this year is anticipafed to be ^tbout-Sl 5, 000. 00. 
The following are some 'specif ic examples of how 
the information retrieval system has been used. 

I, Inventory Data^ , 

Each spring, immediately' before inventory 
time, an inventory print-out is requested for 
each school. This tells the school exactly the 
number of books, filmstrip's, or cassettes, etc., 
they should have in each Dewey dlassification, 
lOO's, EOO's, fiction, etc. On request they have 
also received a shelf list print-out of specific 
sections to be used during inventory time. 

Z. Book Catalog, 

Three buildings 'have made use of book cata- 
logs. The book catalog takes the place oT the 
card catalog. It is an alphabetical listing o£ aUL 
the material' In a single media center. Thet>r.int- 
out is arranged in three sequences; author, title 
and subject heading. The advantage of the book 
catalog is its mobility. A student or teacher 
could take It to the classroom or have a section 
of the catalog for use and not be limited to us;ng 
it only in the/library. The disadvantage of the 
book catalog Isjkeeping it^updated. In our disu ^ 
trict library, material ordering is contihuous,^ 
so the media center receives new material 
weekly. 

3. Building Closings ' 
Last year the district closed two elementary 
school buildings. To assist with the*redlstribuA 
tlon of the library material we requested special 
print-outs. The pri^it-outs showed us which of 
the buildings receiving the relocated students 
had the same material the closing schools- ^ 
We were able to identify specific Items whlch.the 
-closed school had and the other media centers ^ 
dldnU have. This saved us a great deal of time 




and we did not have to rely on the librarians de- 
ceiving ^he -material to try to recall which of 
the ;naterials they already had, nor the jieed to 
bring the shelf list from their building to the 
closing building. 

4. Printed Catalogs 

A number of printed catalogs have been 
developed^ 

a. 16 mm Film Library Catalog ^ i 
Eyery year we request a print-out by 

title 3Ltif\ subi^ct heading of all the 16 mm films, 
in our pistrltt Film Library. This is sent to 
the printer to have 1500 copies made. These 
^copies are then distributed in quantities to the 
schools. A teacher can then take this print-out 
of the films to their classroom or home for 
planning purposes. ^ 

b. Professional Library Catalog 

This ii handled similarly to the Film 
Catalog with a wide distribution. The teachers 
involved in college courses can carry this cata- 
log with thqm to determine what the district has 
in the Profe^sTmi^l^ Library appropriate to their 
classes. 

c. Art Catalog 

After the retrospective cataloging of all 
the art material in all the media centners last 
year, an art print-out by school level was re- 
quested. This print-out was ^Jrinted and distriu-^ 
ted to all art teachers and librarians in the d{^^ 
trict. ' This allows any elemejjtary art teacher 
(or junior or senior high) to know what is avail- 
able from any other elementary School in art. 
Next year the music and physical education teach- ^ 
ers will have this type of print-out available. 

5. E?Hra Catalog Cards 

Ana>mber of schools are requesting extra 
catalog ca!k:ds. One* high school building has y 
satellite libraries (resource ce*i%btrsj in scien^, 
foreign language an?i home economics. By re-> 
questing extra cards they can have a card in the 
main library's card catalog ca^e and one in the 
satellite library. Two buildings receive cards 
for all 16 mm films so their card catalog will 
reflect the holdings in the Film Library. In the - 
case of vandalism or disaster, the computer 
could replace an entire card catalog. ^ 
\ ^ 

6. Bibliographies' 

Several times a week librarians request 
individual bibliographies. The debate coach 
may want to know all the material in all the high 
schools on a topic, or a teacher may be writing 
a new ufiit on a selected topic and will need to 
know all of the available material on it in their 
own media centers. Every request reflecW a 
different need. 

7. Union Catalog 

" A union catalog Is a catalog of all the ma- 
terial in a selected group of llbjjjpries. The 
first union catalog In Shawnee Mission was of 
the high school library resources; this In- 
cluded material in all five of the district high 
schools- It was arranged with author, title, and 
subject heading indexes. The catalog Is avail- 
ablo In the libraries so that am^Jnterested stu- 
-dentX teacher, parent, ordlbrSrian could con- 
sult at to determine the location In the distrlcto^ 



needed resour<^Qs. Essentially /i this function, 
from the user's viewpoint, is as if each high 
school had *a card catalog of the material in all 
the district high schools. It promotes greater 
utilization of material, providers the students and 
teachers with a o^ie-stop source of all the mater- 
ial in the district's high schools, reduces the 
need for all the schools to buy exactly the same 
materials, thus reducing duplications and. im- 
proving the buying power of the district, and 
would promote interlibrary loaning of materials. 

Preliminary cost quotes for such a catalog 
in-pag»er form ranged as high as $10,000. We 
were a&leTo reduce this to less than $500 by 
going to a union catalog on microfiche. This 
cost was considerably below the $10,000 cost. 
The fiche are made directly from prepared 
computer tapes by a locaKcompany. 

The original microfi^:he packet included 61 
microfiche. This represents about 78, 000 unique 
titles in three arrangements; author, title and 
subject heading. There are 27*fiche for' the title 
index, 21 for the subject heading index and 13 for 
the author index. Each fiche has 260 pages plys ^ 
one index fr^me. This interprets to 7119 pages 
of titles, 5392 page? of subject headings, and 
3351 pages of authors. From the originals, ten 
copies of each packet were made. Each high 
school, the Johnson County Public Library, and 
the Dept. of Library Sprvices will have a copy. 

^H^iro^artely this concept will be used in all 
buildings*i{llt is anticipated that when all of the 
library rei^inirces are vn the data bank, such a 
union catalo^vill be developed to show the total 
holdings of the district. * ^ 

IV. Acquisition System 
Purpose 

The principal function of the acquisitions 
unit is to check, approve, and submit requests 
for books, audiovisual materials, periodicals, , 
and library supplies to the computer or to the 
business office where these requests are trans- 
ferred to purchase orders and mailed td vendors. 
All library resources for school libraries, for 
the district professional library, and for the dis- 
trict film I'ibrary are submitted to the acquisi- 
tions department. Library materials for feder- 
ally funded programs, such as Title II, Right-to- 
Read, etc., in the eleven private and parochial 
schools within the Shawnee Mission area, as 
well as District #512 schools aro also ordered 
and processed. _ . ^ • 

' Requests for ordering library material may 
be made by using one of three rnetho3s--the auto- 
mated acquisitions listing, the library request 
card, or the regular district r equisitipn--de- 
pepding upon the type of material and whether or 
not It is to be processed. ^ 

Automated Acquisition System 
The automated acquisition'system was devel- 
oped In 1973 with the fitst print-out generated 
in December, 1973. The acquisition system con- 
t slsts of a data bank of over 13, 00^0 titles of ma- 
terial that has received f^vorable^revlows Inpro- 
fesslonal magazines or favorable preview by our 
district's staff. 
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Every month two clerks peruse 30-25 pro- 
fessional magazinejs and selection aids. They 
complete preprinted grid forms, supplying speci- 
fic data on each item. These grid forms are 
then keypunched by the Data Processing Dept. 
and entered into the district's acquisition data 
bank. Three times a year the computer sup- 
plies a print-out of all the matei-ial input dur- 
ing that time. This priqt-out is then sen£ to 
the district printing deparment to duplicate 
enough copies to send to all media centers. 

The print-out provides the librarians with 
the following data: 

. a.^ Unique item number 
13. , Physical form code (same as used 
in cataloging system) 

c. Author's name *" 

d. TitU < 

e. Publisher > ^ 
I. Copyright date 

g. Edition 
'f> Suggested grade level ^ 

i. Review Source (K Magazine Titl« 
code: 2. Date of |s|ue; 3. Page of 
review) , 
, J. General classification (Fiction, Non- 
fiction, Poetry, Biography) 
k. Number of volumes # 
L Price 

m. Vendot code and catalo'g number 

Three separate print-outs are generated 
e>/ery four months. One for the elementaVy 
rfthool^, one each for junior and senior higtis. 
These are selected from the data bank by sug- 
gested grade levels so there is some overlapp- 
ing Eiaeh Sprint-out is divided into two sections; 
one for books and one for AV items'. 

The librarians complete a grid-type order 
* form when they want to order from the acquisi- 
£icfn list. On, the order form, besides some 
building ^nd funding codes, they simply record 
the form code, item number, gaantity desired, 
unit cost^ and total cost. This form then goes 
^the Data Processing Dept. for keypunch and 
entering Into the business files. The computer 
generates purchase orders for the material and 
cards showing what was ordered. The cards 
are returned to the ^librarian for the active 
order file. As material is received from CLP 
the librarian can remove the cards for the 
items received. The remaining cards tcllwhat 
is still on order but has not been received. 

Cost Analysis 

The Data Processing Dept. ^yearly cost 
analysis for the acquisitions-system is about 
$6700. Two clerks n^aititain t^je input to the 
fl^^hd one library administrator works part 

^fti^e on acquisition. With the 3-times a year 
printing cost the total annual expenditure Is 
loss than $20, 000. \Mherej^the, system Itself Is 

'<Jnly a little over two years old, it is difficult • 
to determine the cost savings related to the 
system. Mpst librarians ar,c'jiow using the * 
system t^ a great extent, but it is anticipated 
that this utilization will continue to Increase 
significantly lft»the next four years. ^ 



V. Magazine Ordering System 

The magazine ordering system was develop- 
ed two years ago. It was designed to simplify 
orders, make them more a^ccurate, provide data 
for bidding procedures, and create a master list 
of who is receiving what magazines in the district. 
The i3istrlct's media centers subscribe to more 
than 600 different magazines. Most elementary 
schools order 30 to 50 titles, junior highs 50 to 
150 titles, and senior highs between 100 to 250 
titles. A data bank of these titles and other 
ordering information has been developed. # 

Once a year a print-out is generated for each 
media center listing the roagazines in alphabetical 
orjder by title of those thery subscribe to. In the 
.late winter, the librarian is instructed to delete' 
/any title from this list which is not wanted for . 
the next year, or change the quantity on any title> 
This corrected print-Q|^t, along with another fotm 
listing any new magazine(s) wanted is then s^nt 
to the Data Processing Dept. to update the files. 

A composite list is printed from the correct- 
ed data banl^ and sent out to various subscription 
v-endors for bids. Once a^obber has been select- 
ed, then all of the magazines the jobber can fur- 
nish are so marked in the data bank. Then, when 
it is time to getierate purchase orders', the data 
bank can print tho;ie for the jobber and individvslr' 
ones for those going directly to a publisher. i 

A list for the individual medi'a centers is ^ 
sent to them in the Fall so they can check off the 
magazines as they receive them. 

A master print-out is produced u/the Fall, 
als^. This li^ allows anyone to identify which 
buildings have ^yhat magazines. .This helps the 
stud^ts by allowing t\iem access to'over 600 dif- 
ferent titles. Onc^ a title has been identified as 
being received by one of the other media centers, 
it can be handled through inter-library loan. It 
also provides a librarian the opportunity to look 
at magazines or talk to librarians having special- 
ized magazines before ordering for the media 
center. \ ' * 

VI. Overdue Notices 

Several years ago « student in our SM South 
High School wrote a computer program to help 
the librarians. This program was designed to 
print overdue notices upon request. South has an 
enrollment of over 2000 students who actively 
use the library, so there Is a great deal of paper 
work related to writing overdue notices. One of 
our other high school libraries will also be try- 
ing this program this semester. 

Basically the system works as follows: 

1. The librarians have a complete deck of key- 
punched cards on all students in the school. On* 
this card Is the student's name and room location 

.for a special class hour. (2nd period at South) 

2. As the student's materlarbecame overdue, 
the regular check-out card Is matched with this 
keypunched card. 

3. These keypunched cartis, along with control 
cards, are sent to the computer center as batch 
work. 

4. The computer center generates a number of 
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print-outs from these cards; a. master alphabe- 
tized list for librarians, a class roster print- 
out for second hour by teacher of those students 
having overdue rriaterial, etc. « 
S. These lists are then used by the librarians 
and teachers to retrieve the overdue material. 

The program is designed to provide s«ver- 
51 different types of print-outs and notices as 
required. This system saves the librarians a 
great deal of time and provides for more ac- 
curate record keeping. 

VII. Business Functions 

The district's business and fiscal control 
functions are totally computerized. Purchase 
orders, audits^ ahecks,_fitcf\ are all automated. 
Monthly the librarians receive a print-out show- 
ing the budgets and expenditure records. The 
business records interface with the library's 
acquisition and magazine ordering systems. 

VIII. Future of Data Processing 

Cojitinued refinepftent and improvements will 
be madie on the exi^ng data processing program 
As mentioned previously, by at least May,, 1981, 
all of the media centers items will be in the data 
bank. This alone will create a greater demand 
on the retrieval aspects of data processing. 

Union Catalogs 

Once retrospective cataloging is complet.e 
for all media centers it is anticipated that a 
master union catalog will be generated. This 
will list all of the library material in the dis- 
trict and be indexed by author, title and subject 
headings. The catalog will be on microfiche. 
This wiU allow any student, teacher, librarian, 
or administrator in the district to know the aval- 

• ability and location of over 200, 000 unique 
library resources. 

Search and Circulation System 
A program will be developed in the future 
to try an on-line, in-building circulatioiKaud re- 
search system at the secondary school lib^'^^ 
level. This system will provide the library wi^ 
a check-in, check-out system and a search sys- 
tem to locate material anywhere in the district. 
i^Thfrse.pj^tedures >viU^allow the students and 

• teachers to ask thelibrarian for a specific re- • 
source (book, film, record, tape, etc.) or a 
general area (Kansas government and history, 
effects of wAter pollution, etc.). The librarian 
will immediately request and receive from the 
Qomputer via a terminal In their media center, 
the location in the district's high schools of this 
material and whether it is checked out and if so, . 
who has it and when it is due back In that llbraiy. 
Thit system will also save the librarians and 
clerks time by checking material In and out, and 

•"by printing overdue notices, fine lists, and other 
lists. It will also produce statistical reports, 
circulatiotfrecords. Inventory records, data 
far state reports, specialized circulation data 
which will assist in accountability and utiliza- 
tion of material* arfd services, and for pur- 
chasing and selection activities. 

• 

16 mm Film Scheduling Program 
The district's Ib^nm Film Library has 




grown during tfie past years from 700 films to a- 
bout 1800 films. The scheduling has grown from 
12,444 films booked in 1971-72 to 25, 000 films, 
booked during 1974-75. A disti-lct the size of the 
Shawnee Mission Public Schools could easily ef- * 
^fectively use over 3000 films. '>The bo6king 
schedule v^/ill continue to increase over the next 
few years. It requires a g^eat amount of person- 
nel time to book these films and eend confirma- 
tion notices back to teachers. It is anticipated 
that eventually a computerized program will be 
developed to handle this booking, providing a ^ 
speedier, m&ro accurate, and time-saving 
system. ^ 

Summary 

As knowledge and information expands, so 
must the libraries in a school district. It will 
not be sufficiejat to have access, only to the ma- 
terial within yfW own school library. The stu- 
dents and the tcAichers must be able to "tap" the 
resources of all libraries in the dfstrict, other 
libraries of all types in the immediate region, 
state libraries, libraries of higher institutions, 
and even data bases at the national level. Re- 
trieval capabilities and interlibrary loan activi- 
ties will increase significantly through inter- 
library communication systems soch as compu- 
terized search and circulation programs, getting 
all library material in all libraries in the district 
in the computer's bank through retrospective 
cataloging, union catalogs of all material show- 
ing location which any student or teacher can' 
look at, and improved inter-library loan pro- 
cedures. 

It is hoped that someday all of the library 
resources in Johnson,County might be linked by 
computer. This wo^ld mean a student, teacher, 
or patron could go to a library and ask for a book 
or other material and the computer could immed- 
iately determine if the material was available in 
any of the school libraries, Johnson Cqunty Pub- 
lic Libraries, or the Johnson County Community 
.College library. Ultimately this network could , 
include other agencies (such as universities, 
medical libraries, etc. ) in the state of Kansas, 
in neighboring states, or even in the nation. This 
type of service would make greater utillzatiopof 
resources^, save the students, teachers, patrons . 
\rno, ^d improves the buying power of the librar- 
lei^^ reducing the need for a large namber -of 
materi^l.dupllcations, thus giving the taxpayers 
more for tholr money. 
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ABSTRACT: A framework for teaching teachers 
i,s presented which exeunines a methodology to 
knowledge and skills to use computers in an 
✓ 

Computers have begun to play a 
^greater role in education. Results from 
the last bien(iial survey of programs and 
practices conducted among public school 
sys'tems in 1974 by the National Education 
Association (NEA) illustrates the trend^ 
of computer use in education. According 
to this survey, comprising a seunple of 
839 school syaCems, it is estimated that 
about 20 percent of the school systems ^ 
provfde compuiers for student use in 
some of their ' Schools* This is an 
increase of t^n percent from 1972. Some 
^of this increase may well be attiJ.buted , 
to the advance of te*chnology both from a 
hardware and software point of view. 
The development .of microp^rocessors, 
minicomputers, and the associated soft- 
ware « package^ related directly to educa- 
tional applications has played a major 
^role in the .introduction of computers 
^ into the public school curriculum and the 
public school administration. Admini- 
strative uses of the computer such as ^ 
scheduling, budget reporting, and 
record-keeping have been widely devel- 
9ped and are very common. TheT devel- 
opment of computer applicatlond^or in- 
struction has not progressed as riMwiily. 
According to some xCducational observers, 
the use of coro^utei^^ in the public 
school curriculum haa been-'^f^mewhat 
injxibttid by 1) the number of teachers 
trained to use computers in the clatjs- 
room, and 2) th^ coat of equipment. 
With the trend toward miniturization^ 
qf components, the cost of equipment has 
been decreasing. Although still impor- 
ta;it^ cost has become in some areas less 
of a consideration. The major area of, 
concern is the training of teachers. 
|is computers come to be introduced more 
into the public -school aystems, teachers 
will be expected to utilize this "n«w" 
technology for classroom instruction". 
Presently, there appears to be an inade- 
quate program in computer education for 
teachers during pre-service training 
(1, 2 ). . 



about computing is described. An approach 
provide teachers with the necessary 
instructional environment. 

Certain barriers or obstacles seem 
to exist which inhibits the learning of 
computing. 3y removing or min^izing 
these obstacles, learning computing 
should be facilitated. 



What then are these obstacles or 
hinderances?i^One of tte major hinder- 
ances seems to be th^^ translation of 
jjnefs thinking abDut a problem into 
^symbols and statemer^ti| understood by the. 
computer. The student is suddenly 
required to pay close attention to 
detgiils and to be= rigorous in hfe think-^*''' 
ing about the steps' that will solve a ^ 
given problem. Associated very closely 
with this phenomenon "is the often 
expressed thought that mathematics is a 
prerequisite to learning ^computer pro- 
gramming. A student needs to be assured 
that his present knowledge is adequate 
for learning computing. The presentation 
of material can avoid many fears by 
drawing upon the experiences and knowledge 
th^t students already possess. It is not 
necessary to speak in .detail of the inner 
workings of computers or the binary system 
of ntimbets. The beginning student need 
only -know that he can commuhicatie his 
problem to the computer through the use 
of- a high-level language. Itf cajn be 
further explained to the student! that 
iN^ranslation or interpretive process 
takes i>lace to jaake his programming 
statements understood by the machine. At 
this stagfe of learning, simplicity is 
important. ^ 
, « 

Another common fear has been the 
"device replacing teacher" syndrcxne. 
Although, probably not as prevalent today, 
this fea'ir nevertheless seems tp operate 
as a threat or barrier to learning. Much 
of this attitude could be dissipated by ' 
demonstrating how the computer can be 
utilized in the classroom as an instruc- , 
tional tool. The computer, it can be 
explained, is one of many technologies 
that can be used in the course of 
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instruction • Other technologies such as 
filmstrips, overhead projectors, video 
tapes, and movies are now in c omm on use* 

Being cognizauit of student anxieties 
eJ^out computing and minimizing these 
hinderances represents the first step to 
teaching teachers computing • Learning 
computing is somewhat similar to the 
learning of a foreign languag^r Certain 
basics such as noun and verb forms roust 
be mastered befo3;e the more complex 
language structures can be learned* In 
programming, the basics include reading, 
writing, and arithmetic* « Once these 
elementary concepts are imderstood and 
the student has had sufficient time to 
test his )cnowledge using these Con- 
cepts, then he may progress to the 



more difficult concepts such as decision- 
making, iterative processes, and list 
processing* Figure 1 illustrates this . 
flow of learning* ' / 

As with any other course of instruc- 
tion, there are certain objectives which 
computing instruction should meet for 
teachers* The minimal objectives are 1) 
to identify those areas where the com- 
puter can be of assistance in the learning 
process, and 2) to provide teachers with 
the knowledge to select between the proper 
and improper use of ccctputers in the 
classroom* 

Some' of the emphasis in teaching 
centralizes iipon the questions of what 
should be taught (content) and how should 
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it be taught {methodology^ • The content 
of an introductory computing course 
should consist of three major coir5X>nents : 
1) a survey of computer applications, 2) 
programmin'g fundamentals/ and 3^) writing 
application programs. More specifically, 
it would seem that the following items 
should generally be included in^an intro- 
ductory course: 

-A survey of computing applications 

tQ education 
-Sources of assistance/information 

on computing 
-Computing terminology 
-Fundamental programming cbncepts 
-Expo8U3;e to the practical applications 

of computing related to instruction 
-"hands on" experience • 

This list although not all inclusive 
•should nevertheless serve as a basis for 
the design of an introductory course. 
The emphasis that can be given to any one 
item will vary according to the amount of j 
cdurse time available and the amount of M 
importance a course instructor would 
place upon each item. Some of ^ese 
items might be comfortably covered in 
one class ' session. 

After a discussion of computer 
applications, it should be apparent to 
the learner what use the computer can be 
in an instructional environment. The 
student should realize: 

-The computer can be very useful 
as a drill ahd prQ.ctice resource 
thus relieving a teacher from routine 
tasks. ^ . . 

-The computer can provide clerical 
support for individualized instruc- 
tion i.e. .computer-assisted-instruc- 
tion (CAI) where a concept or skill 
is taught (tutorial) . 

-The computer is an aid to test con- 
struction, testscoring, test analy- 
sis, and record keeping* 

-The computer can be used to amplify . 
and extend the curriculum. Many 
more problems can be solved with the 
aid of the computer than what is 
humanly possible in the classroom. 

Another examination *t>f Figure 1, 
indicates what may be the content of the 
second major component of instruction: 
programming concepts. Starting with the 
basics, the student is introduced to 
the concepts of reading/inputing data, 
writing data/information r and per- 
forming computations. Instruction at 
this point remains simple with explan- 
ation being supplemented by many 
examples. The student after completing 
this frame of Instruction should be 
able to write a program whereby dat^f is 
read, computation is gSrform^, and the 
results printed and laBoled appropriately. 
This simple process is floW&harjbtd in.. 



Figure 2, 



FIGURE 2. A SIMPLE PROGRAMMING PROCEDURE 



From the basics, instruction pro- 
gresses to the next level of complexity: 
decision-making. Based upon the com- 
parison of two quantities^ the student 
learns how to inform the computer what 
course of action to pursue. The concept 
of branching is explained with examples. 
Discussion includes spch topics as rela- 
tional operators and true-falsef condi- 
t;ions. After completing this portion of 
instruction, the student should be able 
to write a program to splve the ^following 
type of problem: 

A survey was conducted op th^ «. 
ques'tion: J Teachers need improved 
faqilitie/ to do a better, job. 
Agree? Disagree? The agree 
responses are coded rfs l*s and tWfe 
disagree responses are coded as 2's. 
Write a program which will total 
all the agree and the disewrree 
responses for a given set of data 
and display the results appropri- 
ately labeled. 

the student can now writ§ simple p'ro-^ 
grams with some amount of flexibility. 
The next step of learning will* include the 
concepts of looping (iterative processes) 
emd the processing of lists. A dis- 
cussion of iterative processes should 
include loop construction, transfer of 
. control (decision-making) within/outside 
a loopA use of the loop index, and types 
of loops. In discussing the concept of 
.a list, one should include list char- 
acteristics, the pur;pose of 'lists, and 
the specification of lists' in a program. 
Here the student learns about repetitive 
processes and the procedures to effi- 
cently program such processes. The 
student also learns how to set aside 
storage area in his program and read 
information into this storage area > 
through the use of 2^ loop. Once an array 
of information Is established, the" student 
learns how to manipulate this information 
in many different ways* Thjs^ different 
functions of lists ahd how lists can be 
used for instructional applications are 
explained in detail. | This information, 
for example, may be {sorted or rearranged 
in some other manne^ or may be searched 
for a partiTcMlar ^al{ie or -piece of 
inforroation. After completing *his level 
of instruction the student should be able 
to write a program of the drill and 
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practice type. , An example of a 

problem, and the expected results follows: 

Write a program to drill a studfent 
^on" i:he capitals of ten states you • 
^ select. Use the following pro- 
cedures for giving feedback and 
keeping score: 

-If the first response is correct, 
give the learner two points, 

-If the first response is in- 
correct, 'print "a message such a«, 
"WRONG, PLEASE TRY AGAIN", 

-If the learner does not respdnd • 
correctly after the second try, 
thSn no points are given, 

. Print a message "suqjti as,. "THE 
CAPITAL IS" followed.. by the 
correct response,' 

Print a message' indicating the 
score attai^ij^. if, uKe^ student 
attains a score great<^ than 85, 
then print a rewarding message such 
as "YOU HAVE. DONE WELL" followed 
by the learner* s-.riame. If the 
score does not meet this perfor- 
mance^ then print a message 
informing the learner that he 
' ' should review. 



» sample session follows illus- 
trating how the output might 
appear, 

^AT IS YOUR NAME? 
7C0NNIE 

jiame the capitt^ of the following 
.States 

new york?albany 
arizona? phoenix 
michigan?lans1ng 

. 0HI0?T0LED0 
TRY AGAIN PLEASl? 
0HIO?C0LUMBUS \ 

VERMONT?MONTPELlER ^ 
IOWA?DES MOINES* 



JOUR SCX)RE IS 92 • 

^iOU HAVE DONE WELL CXJNNIE 

DO YOU WANT TO TRY AGAIN? 

YES OR NO 

?N0 



TYPE 



The next leve,l of programming 
includes the concepts of functions and 
subprograms. The student learns 'what 
comprises functions and subprpgrams, 
their use, aftd how they i are specified ^ 
in a program. After a discussion of ^pf^ 
'fxmctions and subprograms, the student- 
should be prepared to begin pro- * 
/gramming instructional applicatiohs. 7 
Relevant progi^ams would include ^ 
problem-solving, CAI, or simulation 
applications. The student should now ^ 



pe able to des}.gn a drill and practice 
•unit or a simulation. If a drill and 
practice unit i^ progr«uTimed, the student 
should meet the following. criteria: 

-The program should b^ applicable 
to the student's discipj.ind, 

-The learner's name should be 
requested. 

-Unanticipated responses as well 
as anticipated responses should 
he accounted for and the appropriate 
feedback Should be provided,' 

-The learner should be provided with 
a score or^ percentage correct based 
upoh the performance.. If the 
4 learner •s^perfqrroance is telow an 
acceptable score, then prescribe 
further work. If the learner's 
performance is acceptable, then , 
provide positive feedback. 

The foregoing discussion has focused 
primarily upon content. From a methe- 
dological point of view, this instruc- 
tional approach has assumed that the 
teacher wild be working in a timesharing 
or interactive computing mode. As paft'of 
the learning process, this approach 
expects students to begin by actually 
participating in a terminal^ session. In * 
the ^irst assignment, a student should - 
be confronted, with the experience of 
interacting with a computer. The student 
a*fter a formal Orientation and demonstra*- 
tion of terminal procedures should sit at 
a terminal and participate in a CAI ^ ^ 
session or a simulation, ..Through ?hands 
on" experience, a student should quickly 
realize the uses and the benefits of 
computers. It is ^important that students 
achiove success in interacting with .com- 
puters very early in the learning pro- 
cess , 

To enhance the communication of pro- 
graunming concepts, it is suggested that 
the full-range of audio-visual media.be 
employed wherever possible,- This would 
include overhead projection, video tapes, 
terminal demonstrations, and other media, 
"t^oolproof •* terminal exercises that 
demonstrate concepts should be tegularly 
assigned. Included in the classroom 
discussion should be sh^ftt-^nfobleras/ 
exercises demonstfatiifg various concepts. 
Students should Be ericouraged to woWwith 
others in solving problems during cla3s 
periods, • 

This course atructure'^and approacH 
ha*8 been designed to iiitroduce teachers 
to computing without the necessity of 
expending a great amount of time and 
effort. Acquired expei*ience with this 
approach has demonstrated that it is 
feasible, practical, and can be accom-* 
plished in a miftimum number of fclassroom 
hours of instruction, \ 
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, COMPUTER SUPPORT FOR A'- SYSTEM APPROACH TO INSTRUCTION 
-PROBLEM STATEMENT AND DATA ENTRY TECHNIQUES 

i 

by Eugene A. Collins ahd Dean C. Larsen » 



ABSTRACT. The Jefferson County Public School Sys'tem with the National Science Foundation is conductfnjg 
a study which implements a, digital time-shared computer as support for a systems approach to ins-truction. 
This Study presently involves one elementary school' but will support a 'total of 13 schools in the future. ^ 
Tne computer suppprt includes computer generated criterion test forms. The Items for the test forms a^e - 
randomly selected from a tesj-item bank. Additional computer support includes the use of a remote docu-. 
ment reader for test sconng'and updating student academic records. The records are updated using test 
results and teacher certification of mastery. Data from academic records is being studied to determine ^ 
its usefulness for formation of instructional groups in a multi-unit elementary school. The design of ^ 
feedback reports to teachers i-s being studied in order to optimize the report's usefulness in instruc- 
tional processes. The effects on teacher's roles caused by the computej- support of an existing instruc- ' , 
tional system, are being observed. 

Many programs for indivo dualizing education have been developed during the l3st ten years. An outcome 
of these experiences has b^en the identification of a problem which is common to all attempts to individ^ 
ualize. The pwblem is one of data management. At any given time students "in a particular group may be 
at fiiany different places \r\ the curriculum, each student in the group having different learning needs. 
Information about Uie learning needs of each individuaMs-A>suaUy«-avai Vable. However, -techniques have 
not been developed for effectively managing this information for optimi^g"^^ instruction for each 
student. ^ 

This paper is a description of a National^ Science Foundation^ project located in the J^f^rson County 
Public Schools, Jeffersan County, Colorado. The project is designed to deVelop computer support for an 
existing instructional management information system in schools attempting to individualize instruction. 
The primary purpose of the project is to develop the data management techniques required to group stu- 
dents with similar learning needs. , . 1 

The Jefferson County Public School district has definecf curriculum in language arts, mathenptics, and 

reading by stating explicit student^ performance objectives. Procedures for monitoring stu<fent progress * 

against these objectives have been Identified. The programs have been implemented through^grade 8. In 

mathematics, criterion test items have been developed parallel to the objectives. Levels oi mastery have 

been established. Dependency hierarchies between objectives have been identified, and multjple learning 

activities have been designed which relate to these objective^. 

Teachers are keeping records on individual student performance against the, objectives. However, these 
records on individual student performance are not generally used as a basis for selecting appropriate 
learning activities. Furthermore, in most schools, the formation of student groups foV instruction is 
not accomplished through i/se of past performance records. That is to say that student records -are not 
widely used as input into the formation of groups of students for instruction or into design of instruc- 
tional strategies. As a result, effective grouping of students for experience with appropriate learning 
activities dO€|s not always occur. Students may be provided^ins-tructlon In skills which they have previ- 
ously mastecfid or for which they do not have the i^ecessary prerequisite behaviors. . ^ 

While sy^ematic diagnosis of student's individual needs may occur, this diagnostic ifl^'ormation is not 
always available in a useable form when students are grouped for instruction. This is due primarily to 
the fact that instructional decision makers have too much data. They do not have the procedures nor the 
time necessary to process it. , • . ^ ♦ 

Whe work reported in this paper was performed pursuant tt) Gr^nt Number GY 11109 with the National • X 
Science Foundation, ExperTffifental Programs Group, OffTce of Experimental Projects and Programs. Any 
opinions, findbtngs, conclusions, or recOnmendations expressed herein are those of the authors and do not 
necessarily reflect the views of N.S;F'. * , 
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The project described here is developing, a solution to this problem. Funded by Thje National.Science 
Foundation Office of Experimental Project;?' and Programs, Experimental Program GfOuff, the project is ti- 
tled The C9mputer Support for a Systems Approach to Instruction, A computer system which will allow in- 
structional decision makers to process records on student performance is^ presently being developed. The 
outcome of the application Of this system will be the formation of student groups for instruction. 

The system is being devefoped on a Hewlett-Packard 2000 series, time-shared computer. Communication a nd 
^ remote data entry are handled over telephone lines with a General Electric Terminet 300 terminal and 
Bell and Howell Mark Document ^Reader. * 

Development is taking place at Normandy Elementary School . After refinement and field testing the system 
will be installed in a total of eleven schools in the Bear Creek/Columbine Area of Jefferson County. 
Thqse schools will be added in two waves over a period of four years after the system implementation is 
completed in the development schooT. - • 

The system is being developed in a manner which will provide for the maintenance of performa/ice infor- 
mation on children in any discipline. Currently, mathematics performance records of students in the 
development school are being managed on the computer. Four years of performance history are being main- 
tained on each individual. In order to insure that these performance records are currenti two system 
componerjts^for updating them have been developed. ^The first component provides update by application oj, 
criterion rS^^eaced testing for mastery. The second component allows update by teach^er certification. 

When teachers are interested in testing students over any subset of objectives, in mathematics, they call 
for a test using the test generation subsystem. This subsystem generates a recipe for the test by ran- 
domly selecting items for each objective from an item bank. The itens bartk contains approximately 80* 
multiple choice items.. The recipe produced by the compjjter lists the items to be used on the test,^the 
order in which they are>to appear, and the cbrrect answer. Wulitple choice il:ems are listed first fol- 
lowed by Items requiring more-detailed studeht responses. /Tbis recipe is used by an aide who constructs 
the test by drawing items from the item bank^ 

'1 1 ! 

The decision has been made to t/se only two testi items per objective. • The criteria for mastery is that ^ 
both items must be ariswered coj:rectly. I I & 

The test is adnfinisfered to students. They resbond to multiple choice iAems on an answer card shown in 



ERIC 



figure 1. Studerjts respond to non-multiple choice items 
Items are then graded and transcribed to the aniwer card 
th^ test recipe. Thislkey is enftered through tile mark do 

carps. Grading of the test >s done by computer! The student's acajdemic performance records are auto- 
matical^ updated by thp system. A summary of tfest resulps is printed 'out for teacher analysis 



)n their test pap^. These non-multiple choice 
)y adult. An answer key is filled out from 
umept reader along with the student answer 



The advantages of this procedure are numerous. First of 
objectives are tested together. Random selection of item 
one form of each test can be generated, and there is no n led for computer storage. of the answer keys 
Automated grading and record update is provided along wit i enough human interface to allow fOr non 
multiple choice items* 



ill, ther* is total flexibility regarding which 
from the item bank is a 1st) assured. More than' 



A second component of the system allows for update of student records through teacher observation and 
subsequent certification of mastery. This is a necessary component of the system because many mathe- 
matics objectives through the second grade are not appropriately meas:ured using paper an<j pencil. Stu- 
dent behavior described by these objectives includes verba'l response and application of manipulative 
devices. These behaviors musj; be observed and recorded by^the teacher. Furthermore, this teacher obser- 
vation and certification is an appwropriate assessment of student behavior beyond second grade. 

The need to update records by teacher observation and certification is being supported by the form shown 
in figure 2.' The form is initially blank on the left side! The heading, containing student name, system 
number, mathematics level, etc. is printed by the terminal. Forms for each student in a^particular group 
are kept together in a folder. Teachers observe individual student behavior. Mastery of particular ob- 
jectives IS then certified by marking the* appropriate code on the cQmputer readable part of the form on 
the right side. These forms are read periodically by the mark document reader which has the capability 
of reading both 3U" and 8^" width forms. Student records are updated accordingly. 
» >• 

As m'entioned Before, the form in figure 2'is Initially blank on the left side. Alternative types of in- 
dividual student profiles are being printed remotely by \he termifial In this space. The project staff 
and teachers are working together to determine which forms of the individual student profile are most 
helpful to thQ teachers in Supporting their efforts in obferving students.' 

Now that many of the mass data^entr^ problems have been solved, the primary task of^ the project is being 
pursued. The ^ask Is to 1ncrease'tlit.>U6eabi 1 ity of the data for teachers,' usit^g It to form instructional 
groups. The development school, as /dl as many of the other project schools, - is Implementing Individ- ' 
uaMy Guided Education. Under-.-th1s nodel, schools are organized on a multi-unit basis. Each unit Is ^^ic>.' 
composed of IGO to 150 students ^w1t(/ up'^o thre^.^ars age difference. The formation of such multi-age. 
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units increases the heterogeneiv^y of the unit with regard to' student's previous background artd experience. 
This increased variety enriches th^ learning experience which can be tirought to any instructional subgroup 
formed within the unit. This heterogeneity is important to learning in any discipline. 

Within a multi-age unit, instructional silbgroups are formed in the skill disciplines on the basis of 
cormion, student learning need. These skill disciplines are typical ly mathenratics, reading, and language 
arts. Based on their observations and previous experience with students, teachers identify-clusters of 
objectives for which they think instruction is needed. Student's needs with regard to these objectives, 
are assessed. iT instruction is needed, appropriate student groups are formed. ,^ 

The multiTage unit organization along wi^th the procedure for subgrouping on the basis of 5^111 needs has 
the following advantages. Some of the heterogeneity present in the total unit is maintained in the in- 
structional subgroups. Furthemtore, variability in student achievement in any sut)group is reduced. 
This allows the opportunity for planning an optimal learning experience for eaclr; subgroup. 

Student learning needs related to any cluster of objectives in mathematics are assessed in two ways, 
through diagnostic test procedures and through the analysis of historical records of student performance. 

When a team of teachers feels the need to regroup within their unit, the objectives for potential 
instruction, during the next instructionafl period, are identified. A pretest constructed over pre- 
requisite objectives using items from the test item bank. After administration of the pretest, the 
results are analyzed by the Ceachers and the project staff. Subsequently, student subgroups are formed. 
The criteria for this subgroup formation are recorded as precisely as possible. ^ 

,While the teachers are provjding instruction, the project staff uses the* computer to probe the histofical 
performance data in an attempt to form similar groups. The stated grouping criteria are used as a guide 
for developing alternative algorithms for subgrouping. These algorithms are programned and' executed. 
The composition of the suggested groups which result is then compared with the compo^tion of groups , 
actually receiving instruction. In the future this procedure will be repeated regularly. The desired 
outcome is the discovery of generalized algorithms for. probing historical performance data to prcrauce 
instructional groups which are- similar in composition to those obtained through pretesting. 

Any successful application of comouter technology to support instruction requires attention to the inter- 
face problems which exist Between the user system and the technical system. The level of technical 
literacy on the part of educators is^of primary concern. They need etipugh technical literacy to under- 
stand both the power and limitations of the ComputerLand to Qmmunicate precisely with it. The required 
level of technical literacy may be different for each educator. This depends on the educator's role la 
the schools and the person's relation to the information system. Th? various roles and relations must • 
be observed. The necessary level of technical literacy must be identified and provided. 

Thp problem of understanding and precisely modeling the users original information System is of equal 
importance. Technical staff members must recognize^their role as one of modeling the users information 
system rather than redesigning it to meet some external constraints. Only if a f^recise computer model 
is produced will meaningful computer service be provided. 

These interface problems are much more severe than generally recognized. They are soJved only through 
well plann£d procedures which increase person to person communication. In order to increase person to 
^person communication, the project has chosen the procedure of on-site development and sl(Jw- expansion to 
other schools over a period of several years. 
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ABSTRACT: 'A 2 year pilot study of interactive computing in teacher IWucation began in 197!| at the Univer- 
sity of Iowa-. Included is a discussion of the organization, ccjpperatiaSn of university -unip , goals,^ imple- 
mentation, results, and future plans itivolve<f^ in integrai^g huwan and hardware te^chnology. 



IirTRODUCTION ir. ' ; 

C- F. Hoban, in his discussion ©f.^jie maoageaent^ 
of educational technology, noted "...technology, in 
Its modern usage , Involves the' in5n^gemeht of id^s, 
procedures, money machines and people -in the in- 
^.tructional process." (Hoban, 1968X &ven so, many 
who vould introduce computer based inptruction into 
the curriculum focus on machines and -the ^"technical 
language" /^nd give limited, if any, attention to 
the neey and motives of those they intend to 
serve. /Consequently, meaningful efforts are often 
doomed rtt |kllure even before they are -pp«§ented 
to students and faculty difectly concerned with 
curriculum development. 

The purpose of this paper is, to present, one 
college's effort to aystematicaXiy develop a plan 
to introduce computer based education io the on- 
going curriculum. Through the planning process, 
a concerted effort wai made to integrate hardware 
acquisition and the design of t<!Chnical supporjt 
systems with ^ mQre fundTamental committment to [fac- 
ulty development. Institutional development Was ' 
viewed as the p^(fiit:t of systematic integration 



of c^611ecti'«e contributions of- 'cre<icive faculty, 
staff and :stj;Idf nts ,to riew applications of ideas, 
material aijJ^Sirdware for ^mprgVed learning. 

DEVELOPMENT OF INSTITUTIONAL CO>fc[TMENT 

,An analys^s^^f. c'ompuCdng apblicapions at the 
University of lUta conducted inj 1972 revealed that 
only 15 percent of total comp'ut,'e,r usage wap com- 
mitted to instruction. Most iijstructional appli- 
cations Vere designed to teach (research methodol- 
ogy with practically none targetedTor undergrad- 
uate instructign. A preliminaty plan was devel- 
oped to introduce interactive /competing in iinder- 
graduate cu^riculS throughout ithe campus. The 
preliminary plan included x*ec6gnition of the crlt;^ 
ioal necessity for attentl<^rt» to eadh of* the fol-, 
lowing support systems:^ ^ '.^ . 

1) A s'ystem to fa<filitate faculty involvement 
in tt)e planning and development process, 
^ 2) A systenf'to provide necessarp^ technical 
support, 

3) A system to provide essential communication 
» among participating groups and activities. 
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and, ' : 

4) A system of hardware and facilities to^.s^" 
port emerging development and^ttnplenenta- 
tion. 

' ♦< , 

Th^ initial plan for an experimental ffiroject 
< was 'Visibly supported by the Director of the Com- 
puter Center and the Vfce President for Educatlon- 

>al De^yeloprqeht and Rei^^ch. Together, they ac- 
knowledged Ihe^ need for collegiate faculty Involve- 
ment and support and announced a commitment to the 
acquisition of an integrated system of mlnl-q^ompu- 
ters to provide hardware support. At ,the same 

' time,, support staff of the Computer Center were ^ 
designated to assist collegiate units' in their sys- 
tems development. (Weeg, 19 7 i) 

Selected colleges were invited to participate 
In the project. If a unit elected to participate, 
the Dean was asked to designate a planning^ commit- 
' tee to wprk with the Office of the Vice President 
for Educational Development and Re^search and Che 
Computer Center. Each collegiate planning commit- ' 
tee was to have responsibility for site planning, 
necessary training and development, and evaluation 
of' pro jecf development. 

The procedure for Initial planning provided 
several essential ingredients for effecting change 
or development in academic organizations. 

1. The proposed project had visible support 
from the University central administra- 

2. .A commitment to technical staff support 
was assured. . . * , 

3. Collegiate sysTems received a commitment | 
of support: from Che dean through the des- 

*ignation of a planning committee. 

4. An integrated system of hardv^re support 
was committed. 

6. A system of communications among university 
administration, computer center support 
staff, and collegiate units was established 
through interlocking planning committees. 

6. Above all, a vehicle for faculty involve- 
ment in the planning and development pro- 
cess was Identified in the form of planning 
committees at the collegiate level. The 
substancjB and direction of program devel- 
opment efforts were the responsibility of 
faculty planning committees, 

DEVELOPING FACULTY Iha^OLVEMENT AND COM>!lTMENT 

^ ' Numerous studies have.shov.'n that the Inclusion 
of those iflost directly affected by plans and de- 
cisions leads to better input and ultimate re- 
sults. Similarly, involved professionals are also 
more committed to goals and programs they have 

[ helped develop. Above all, innovative 'development 
of instructional programs is jaost effectively and 
* efficiently, realized through the effort of those 
vjlth gj-eatest expertise - the faculty. Thus, re- 
sponsibility for early planning effort was assigned 
to representative faculty and staff committees at 
the collegiate level. 

Representatives named to the College of Educa- 
tion Computer Based Education Committee included; 



"senior professor with a strong compiTter 
background ,^ 
0-A senioi? professor with a strong background 
in learning theory and instructional design, ^ 
-A junior professor from secondary education ^ 
with prior experience in computer assisted 
instruction, 
-A junior professor from educational'media and 

instructional development,' 
-A junior professor from elementary education, 
-A graduate student in educational psychology, 
-An undergraduate "Student in elementary edu- 
cation, and 
-An assistant dean. 

-TV 

The college committee was strateglcalj.y select- 
ed to itiplude staff with prior experience in the 
use of the computer, instructional de&i!gn and de- 
velopmeaCr> and positive approaches to Ta'eulty de- 
velopment. 'Also, the two largest undergraduate 
procpams (elementary and secondary education) were 
represented as well as the office of the dean. 
The committee was structured to facilitate comple- 
• mentary input of £;pudentSv faculty, and^ administra- 
tiv'e staff. 

Nominations to the committee were made from a 
list of individuals who had responded positively to 
a survey of interest in computet assisted instruc- 
tion. 

The change of the committee included responsi-^ 
bility for site planning, a plan for systematic 
program development, and articulation of' procedures 
for project evaluation. ""^ 

4 

ARTICULATION OF PROJECT OBJECTIVES ^ 

The college planning committee focused its ini- 
tial attention to the development of project objec- 
^ tives' which were to serve as guidelines for project 
development as well a reference point for coor- 
dinated direction for interested faculty and staff. 

• ' The recognized goal of the experimental project 
was to make a significant impact on the use of com- 
puter based education in undergraduate^curricula. 
The committee recognised that improved Instruction- 
al effectiveness and acceptance of products devel- 
oped would be contingent upon 4eliberate examina- 
tion of each step of the instructional design pro- 
cess. If the total developmentVprocess was to suc- 
ceed, cominitted faculty would need time, technical 
support and recognin'on for instructional design, 
software development and' evaluation efforts. Em- 
phasis was'^lven to the reality that significant 
contribution^ would result fro^ programs which were 
developed from deliberate analysis of * instructional 
needs rather than CAI packages constructed as ends 
in themselves In search of a problem. 

The following general o'bjectives were articu- 
lated by the planning committee. 

1. Stimulate Faculty Development ; Identify a 
core of committed faculty to develop spec- 
ific computer based education' programs and 
provide sufficient released time for ^ro~ 
%xtin^\OiTiTiiTi%y development, and evalua- 
tion. Those selected would become the nu- 
cleus for continuing program development. 



2. Maxtmtze Student Impact : Select and d^vel- 

op prd^ects whiVh could result in immedi- 
ate impact on a^igniflcant number of^ 
' * students in the undergraduate teacher! ed- 
ucation program. \ 

3. Integration of Instructional Media : Explore 

and develop the potential of computer \ ' 
Ijased technology as a complement to exist- 
/' ing medi^^and methodologies. Concurrent- 

ly^) the plan called for establishment of 
a support lab to prpvide continuing hard- 
ware, software, and technical suppor^to 
\ facu-lty and s&udents^. 
U.\ Promote the Acceptance of Computer Based 
Education; Through project design, .field ' 
testing, and etaluapion, provide a sup- 
port base for computer based education 
^programs as effective learning strategies. 

5. Development of Generalizable Products : 

Through all of the above, the College 
c6mmlttee was committed to develop pro- 
grams, and products which could be 'trans- 
fer red and implemented at sister institu- 
tions throughout the country. 

6. Continuing Faculty' Development ; As a part 

of the total plan.^it was intended that 
the total project would provide an iricen- 
tive and example for continuing faculty 
development and utilization of comput'fir 
based 'education ptogXams and materials; 

^ Thes?^ ob j-e^ t i ves were "Intended to* integrate t^e^^ 
commitmenc- to hardware a.ccessibility . faculty sup- 
port and development, and brodd student acccs'sr; 

^MPLE>tEKTATION PLA>^S A^n) STRATEGIES 

Through a statement of prT^fect goals and^ objec- : 
tives, tie planning committee sought. to clarify 
expectations for the project. Still, plans to 
I4entify and assure necessary space, 'hardware, per- 
sonnel and software support remained to be devel- 
oped. 



Physical Facilities : The committee enphas 
three criteria in their plan for ph/sical- fac 
ties for the computer based education project 
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A space* large enough to accomodate a critical imass 
of. terminals was essential. Ef f iciency^of student 
utilization was not to be constricted by thin dis- 
tribution of available hardware. 2X Space for the 
computer based" laboratory should b^^in, ciose prcfk- 
iraity with other elerients o/ instructfonal resourc- 
es in the Learning Resources ^Cent^r. 3) Facilities 
should be provided in an environmental setting 
which would be pleasing to users. . » ^ 
, ' > 

Space adjfigent ^o t*he existing Curriculum Lab- 
oratory was reassigned for use us a ^Computer B^sed * 
Education Laboratory. An interior designer from a 
local arrht-tecCural firm was^ engaged in. a .plan ib 
renovate and furnish the reassigned space. 

^ . ' • .* 

The . laboratory w«ns carpeted, wa^s *and ceiling 
treated with acoustical tile and compKmentary col- 
ors. *and furniture was selected. to complete the de- 
cor. A partition was designed to give privacy to 
a work spnc^ for a lab coordinator. 

Facilities have proven* to be functional and at- 
tractive. ^ 



Hardware : Because the basic support system had 
already been selected by the central planning com- 
mittei and eight terminals Hour CRTs and four 
TTVs) assigi^d to the College of Ed\i<^ation, the' 
primary concern of the college comn^ttee focused on 
proper distribution and functional assignmept of 
available terminal har^lware. The committej expres- 
sed commitment to optimum acce^sability for student 
users, special support for restricted faculty de- 
'velopment, and flexibility. ^ 

Six terminals were clustered in the Computer 
Based Education Laboratory and hardwired to the 
central system. Two terminals were designated for 
faculty dfevelt>pment, and placed in locations iso- 
lated from the lab for uninterrupted faculty use 
(one CRT aryi one Teletype). Thus, no direct super- 
vision or technical support was available with de- 
velopment terminals.' Two of the eight ports as- . 
signed to the college were maintained as tele-ports 
to assure hardware mobility and flexibility. 



During th'6»first7, 
showed that -the facul 
receiving little use. 
CRT Into the student 
variable-speed j^ortab 
which- could be taken 
stratidri and to 'ather 
^courses. The portgbl 
§ubstant£aA use ifi^^fa 



^ear of the project, data 
ty developmet^t terminals were 
It was decided to bring the 
lafb and trade t]ie TTY for a 
le Tijcas Instrument Si tent 700 
into public schools for clemon- 

campuses for e>?tension • 
e . terminal has also received ' 
culty/ development. 



Within six months,' it became" apparent that de- 
velopers would -be able; CO accomplish more if they 
^ere not Interrui^ted by-I«boratory activity. An 
office isol^ated from the* student lab, bub. nearby, 
was designates as a 'development otCide and a CRT 
teriulnal was moved tVprovide a quiet place for 
both student and factflfy development. • 

The iTYs are now over two ancf ofte-half years/ " 
old. They receive one-third to. oncTrhaif the use of 
CRTs. Only people who must have hard-copy are 
willing to'^se the slower rTYs. :.3ecause the TTYs 
have depreciated over half their expected five year 
life, maintenance is becoming a problem. >Also be- 
cause of tiheir sLow'gpeed and high noise ievel^A . 
consideration is being given to replacing tik ti^e^ 
with DEC writers or som6 sl}ntlar device one at a - 
iUrtc over the next^ three years. Some. hardcopy de- 
vices are necessary for developers who i^eed list- 
ings ^and for'u^er^^ho want print^out. Thete' is 
presently no pla^ to replace the Beehive GRfe^t/hlth 
. have provided good service while receiving more 'Use 
tt^ati the TTYs. ■ ' ' / ^ . ^ 

?ersonnei-|Faculty t^esgurces t The comiiiittee rec- 
^^ognized faculty and staff support as being the most 
cruciaj,>slement of their effort, to develop an on- 
going cbmp^utpr Sased education progrniji* Few faculty 
had prior experience 1^ instructional applications 
of 'the dompii^er; software directly applicable to* 
teacher 'Odud^it ion wa^ not immediately available; and 
no systematic program fox* staff developnenf had been 
identified. iThus, the committee outlineii -steps, to: 

1) Create an awareness of the potential role of 
computet based education in teacher educa- 
tion."' ^ ^ ' * 

2) Provide resource support foi? faculty who "ox-* 
pressed Interest l{i the' project. 
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3) PJ^ovide access to available software, and 

4) Entourage recognition for development ef- 
forts.' 

♦ *- * 

A survey of all faculty in the college waS de- 
signed to solicit expressions of interest. Re- 
sponses from the survey w^re lised to identify pro- 
jects which could lead to early success and inple- 
mentatlon. An early effort was made to develop a 
propo^^l for external funding to support project 
development throughout the university. Internal 
resources of the college were ident;ified and re^ 
allocated to support selected development projeijts. 

Probably the single most effective event in stim- 
ulating C.A.I, development was the intercollegiate 
sponsorship of a simulation .designers workshop pte- 
sented in the fall, 1974. Dr. James Bobula o& t^e 
University of Illinois Medical Education Center 
spent two at>i» one-half days assisting designers it 
dey^iopTng paper and pencil simulations. Some ofj 
the paper and pencil simulations were later ex- 
panded and implemented -on the computer. Some of 
these modules become the core r&eearch for Ph. D. 
, disserifations as w§U as other research studies. 

. Other major events include summer grants given 
by the University Computer Center and University 

^Council In Teaching to professors to develop C.A.I. 

-^^acerU;^,. Courses in CvA.l. languages and more 
general. Coraputers' in Education courses have 
aroused intferefit in students which frequently re- 
sults in development modules. Local newspaper and 
rpiversit\Vpre^s releases have*fievoted articles to* 
coruputin^. and its impact on education andf to in- 
diviiiual fa<!ti^!:ty projects being developedlat the 
University. ... I 

\A dly-Ipng faculty retreat was held oflf-campus 
wh'lch focused on the inoovative use of irjstruction- 
al -^echnolo^y. The Dean. encouraged faculty mem- 
bers* to disnijss classes tor the day so all' faculty 
md <it;udent;S vho vit«hed t»5. would be able to attend. 

Continuing incereftt- in computer § imulatioa has 
led to two- additional 4j.-aulation designers work- 
shops pfe^ent^d localiy'Vy participants of earlier 
woricshop^' vho' f5/t>5;ressed into ?OTro .sophisticated 
computer applI\:ati'on tecftniques^. 
^ ' •. 

An information exc^atjge with the other Regents 
institutions led to addiDlona^l interest in devel- 
opment. A similar exchange xt't^ the University, 
of Neljraska at Lincoln should i>lso stimulate *de^ 
velopment . ' \ 

^^heiyt, has been a- great deal", of promotioxiAl ac- • 
tivity in order to create^ an awk^e'ness of cc^nput- 
ing and Xts^ value as ^'irtstjuctional aid. A num- 
ber of \is^t-% and developers ..ire involved and reg- 
ula'fiy utilize'the facilities p^rovided. ^Ac.ten- 
tion fs 'now being t-urned to tho'lse* users 'who are 
interested but who do'^not have the time or exfVcr- 
tlse to complete projects .^^ho are in -^pcBS which 
hJwrmade little^^u^e of compra.Cing. Th'6ri is a ' . 
core of sophisticated -users who .^ssist tieir f ac- 
ulty colleagues with new developnient . Ir| this* * 
wAy, the expertise of a fev^can be vieweian a< 
multiplier chrough'out the college. 



formally recommended recognition for faculty de- 
velopment of software in promotion and tenure de- 
cisions. 

An analysis of our brief history has shown a 
positive relationship between developmental events 
and the incidence of usage. ^ 

Personnel-Staff Support An early commitment 
was made by the College administration and the 
faculty developers and student users. Resources 
were thus committed to basic support positions. * 
The goal of the computer lab staff was to prqvide 
bafeic development support and general continuity 
to lab users and develdpers. 

The initial personnel consisted of a half-time 
graduate assistant, who had five years' batch pro-- 
gramming experience. General direction came from 
the Associate D^dn of the College. Later, another 
half-time graduate assistant programmer was added. 
The two students split the responsibilities into 
promotion-administration and programming. The 
students shared responsibility for assisting lab 
users. 

A .year and a half into the* project , the status 
of both graduate assistd^ts was changed from half- 
time to three-fourths-time. Another half-time 
graduate student was ^rdded temporarily for devel- 
opment of a specific project. A work study pro- 
gramming assistant was added for a summer project. 
Concurrent support was provided for graduate stu- 
dent input as content sjJecialists in selected pro- 
jects. 

After three years the ^taf f was expanded to in- 
ccbporate a full-time coordinator% a half-time 
gtiiduate assistant programmer, a student, program-' 
mlhg assistant, a secretary shared wittt the adja- 
, cent Media Lab, and a student typist alio shared 
with the Media Lab. The coordinator reports to 
the Associate Dean of the College and serves as an 



ex-officio member of the Colle 
Committee. 



;e Computer Advisory 



The Computer Based Education Cqpsnlttec 



Throughout the project, thel advice of C.A.*I. 
consultation at the University Computer Center has 
been invaluable. The Computer Center has also 
provided BASIC and IDF language workshops as well 
^ as other workshops related to computer usage ^ 

As part of the Learning Resource Center which 
also includes Curriculum and Media Laboratories, 
and the Education-Psychology Library, the Computer 
Laboratory is able* to draw on the expertise and 
technology of these resources for multimedia mod- 
ule development. 

The cQmbined efforts of feiany people have gone 
into making the Computer lab operational. 

Software T»he coordinator jof the jConputer Lab 
assumed responsibility for tlje identification and 
preview of software packages external to the Col- 
lege which might prove beneficial in the teacher • 
education program. ' 

N'ew package titles developed by stiudents and 
faculty within the College are listed below. (A 
complete description of projects is available upon 
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request. ) 



DRILL AND PRACTICE 



Macii ' Basic math review and remediation. 
Media - Review questions for old terra and final 
exams . 

Spanish - A brief program reviewing grammar 
, of the present tense in Spanish. 



TUTORIAL 

Systems Apiproach - An introduction to the systems 
dppr6ach as*^i't pertains to education. 

Learning Theory V To teach political science stu- 
dents the irapiications and applications of 
learning theoiy to. socialization. 

Interaction Analysis - Teaches studeiits the cate- 
gory system of Flander*s analysis; applies the 
knowledge usiAg audio tape conversations of stu- 
dents and teacners which students learn to eval- 
uate. I 

Behavioral Objectives - ^A tutorial introduction to 
writing, behaviclral obyjectives. 

Photosynthesis -A tutorial on the photosynthetic 
reaction'. \ 

Math - A series of progrdras on complex numbers. 

Computer Programming \ An IDF program t6 teach IDF- 
users who have had no computer experience.. 

Photography - A tutorial which ^leads beginning stu- 
dents through the meaning and use of depth 
field. . • 

Educational Administration - A tutorial program 
which teaches the basic\ conc|pts, condltiohs, 
formulas, -and types of queues. 

Statistic**^ A teaching raodule which stresses key 
'points In s\«^tistics and allows students to 
build Eampiing distributions using simulated 
data. 

' ' SIMULATION 

I 

Photography - This program teaches students how to 
process Ektachrome film using a simulated dark*- 
room. * . • 

Photography - Instructs beginning photography stu- 
dents In the process of making black and white 
prints. 

Statistics. - Computation modules in Bayesian Stat- 
istics where students input data and are able 
to obs^=rve the results. v 

Computers in Education - Introductory program 
dealing with differerft aspects of interactive 
computing In instruction. 

Math Education -^Teaches prospective elementary 
school teachers the basic vocabulary and nomen- 
clature. . » 

Classroom Test Development - Teaches hbw to compose 
effective classroom tests from a pool of 100 
items. Gives statistics from the test produced. 

Llementary Science - A 4-lesson simulation on force 
and motion accompanied by film loops. 

Delphi Inquiry Method - Program' on the Delphi tech- 
nique With simulation and tutorial modes., ^ 

Sociological Game - GHETTO, where students are pr/s- 
sented situations encountered by ghetto-dwellers 
and try to im^)rove the ghetto conditions through 

^ their decisions. 

Law - Two simulations which teach problem solving 
^in civil procedure. These are being used in 
first year civil procedure classes at'* University 
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of Iowa law school. 
Teacher Education - A set of classroom management 
simulations for student teachers of elementary ^ 
grades. 

Teacher Education - Series of management simula- 
tions which confront science education student 
teachers with classroom situations about which 
they must make decisions. \ 

Counseling - Simulation for counseling education 
students where the student is faced wich a 
client and the student must' make decisions 
about the problems* 

Teacher Education - Questioning Techniqyes —A 
simulation in which student teachers V^sk" 
questions to gain skill in 'achieving teaching 
objectives through questioning. 

Teacher Education - Elementary Math — Elementary 
math simulation — a course to assist teachers 
in diagnosing and tailoring elementary math to 
meet the special needs of the students. 

COMPUTER-MANAGED I.^STRUCTION 

Instructional Design - 12 CMI modules which tes£ 
over readings in instructional design. 
**bomputers in Education - Eight CMI modules which 
survey computer {application in| education. 
Teaching of Reading - A CMI diagnosis and pre- 

script ion reading program. . 
"Counseling - Two testing modules for counseling 
stud^ts. 

Media - Avtest item pool containing more than 2000 
items to generate tejfXs covering a rahge of ex- 
perience in educational media. For IBM 360 
system. ) 

EVALUATION ^ 

Evaluation of the project, and products which 
emerged f rom *the development effort, focused on ^ 
four primaf-y concerns . 

1) Have development efforts resulted in in- 
creased student^use of computer based in- ' 
struction? 

2) How have students reacted to phe intro- 
duction of computer-based units in their 
classes? ' f 

I 3) How can support facilities and services 
be improved? - 
4) Above all, can any measurable indicators 
of improved learning effectiveness be 
identified? 

In the academic year pr^ior to the arrival of 
the Hewlett-Packard interactive system', (1972-1973) 
344 graduate and u^per division students used the 
IBM 360/65 system primarily for research in 11 
classes. The first semester that the H-P was 
available, there were 582 Undergraduate and grad^ 
uate students in nine courses using the H-P syste^.' 
There has been a continual increase 'in H-P use by 
both undergraduate and graduate students. 

Student and faculty development has remained 
fairly constant since the beginning ot the H-P pro- 
ject. Currently thete are 28 .major instructional 
modules completed and three still in development. 

Table I re|)orts t^e use data since the beginning 
of the project. 



31? 



o 

ERLC 



44- 



/ 



< 



m 45 



ON 



00 



u 

C/5 



6C 

m c 



ir. 



< 

t/i 



to 



'I 



>^ c 



.o 



o 
6 



45 



318 



\ 



\ 









n 


n 














Wi 


CI 










V 


o 


. D ^ 










a 


a 












0 


0 


H CI 


n 










>v « ^ 




c 


o 


C 






u Wi CI 




0 


0) 


CI 




> 


CI > 


c c 


i 


u 


-o 


M 


o 


;) M 4) 


C ' 


c 




0 




Q 


O ^ Q 




eo 


CO 


«J 

0) 
























4 



g 



Currently, cunditiuns, costs, and distVubution 
of niaterials are concerns with developers. The 
Computer Laboratory, Is cooperating with CONDUIT, 
an established distributor^ of computer materials. 
CONDUIT is just beginning to handle education^and 
interactive modules. ** ♦ 

I 

It has been the policy of the l^i^b to maintain 
records^of events and/or strategies of promotion 
and not6 the result in terms of operation, 4^^^" . 
opT^snc, and student (is6. For exainple^ the iso- 
lated development terminals intended for 'faculty 
use sat idle mucV of the tim«. The faculty sel- 
dom uirectly programmed ot entpVed their marer- 
idls- Students^.v^no asst'Si^^ed f^^ulty often needed 
prugramming assistance so when- Che devejppment. 
terminals were relocated nearer the lab prus^rar?- 
ming assistants, their use increased. 

Students wj&re exposed to both instructional 
programs and computinj^ as^an instructional aid. 
An dttitudinal survey regarding several facets of 
Che ^.omputer and computer related applicatTions was 
tltub ddriinistered to^ sampling of all. students. 
The attitude survey polled 19 undergraduate and ^ 
graduate students in the summer' of 1974; 1*42 In 
fall, 197:^; and 125 \n spring, 1975-. 

Result^ ot the largest sample (142) indicated 
* that 90 percent ^of the students had never used, at 
computer before. Ninety percent agree that the P 
computer was a us^ui- tool of instruction- Al- 
Cttough 85^ ^aid the^V-i^ad^no knowledge of j»;rograni- 
-iin^ lan^juages, 70'. repgrted that they wculi. like* 
to take more courses losing the computer though not 
neces:5a'rily to program for it. About 50% said if 
a computet we're available, they vould use it In 
their own[ teaching. Howeyer, 50* fel<* that a com- 
*pater course should not "be required for either a 
degree .or teachdr certification^ 
I 

Process evaluation of plans fofc .support facll* 
^ici^!s and services ^As conducted throughaJt the ' 
fJruJect. Thus, developmental terminals were re- - 
located CO better serve us e^ needs pnd st^ii al- . 
•locations were adjusted accfcrdinj; to emerging- de- 
mand . * » ! ' . - * 

hvaluaCion of co^nitiv^ gains ao4^ learning »ef- 
^ectlveness has been a structured objective within 
all projects. One complete module rcporti* results 
of a teultimcdla series which teaches Flanders' 
Iftterd<.ti<>n Anal-v.sis co-u^crgradwt^ social .stud- 
ies ntthods student-?.' This moduTjR shows that of a 
. posbible 22 poSt*tcst rcsponsttt. the experimental 
computer group (N"14) ha^ six>perlcct scores and 
five scoring 21. The lowest- score Jas 19 •• the 
control ^roup CN*13) matched to the experimental 
group on ACT >ajW cumulative grade point average, 
the high score was 20 and th§ low was D. Time 
compression was also^ significant. The experdmen- ^ 
t^l group spent 2.2 hours In Ipse ruction, j:he 
control group 3.3.^ It also appears that students 
with low ACT and CPA generally performed better 
vim C.A.X. th^n wlith the traditional netho4^. ^ j 

.Similar results were obtained in a serie.*? of 
foUr film 3»oop and comfkute^' Simula t long to teach 
the relationship force and rjrtion according to' 
Newton *s Second Low, Thrjee groiips wct'c used. 0n« 
Rrvup' viewed the fitos an<3* used -tthe compute?? simu- 



lation, a second group studied the same material 
using non-computerized simulation materials and a 
science lab, artd a third received the usjial lab and* 
class instruction. The Computer group spent a mia- 
iraum of one hour and forty minutes on the termin- 
als. They did no outside work on the material. 
The student-who had benefit of the lab and.simula- 
t^n material spent. -^ix forty-minute periodsion the 
material plus 43 minutes for each student outside 
class. The regular ^lab group spent between nine 
and eleven Ciftyrminute perU'o;ls on the material 
plus twentyi-.-«[o^ bh'ir^ jginutes.on.-^pmewocK. The rap 
tio 6f concept.* learning for'^he c;q!nputer. group com- 
pared to the* Tegulfri:- 'l^b -g^t'otxp^ was^ 8:3 ^6. 1 . The 
group which 'r^eived vacces-s" eo--t^* .simuXet ion mat- 
erials and fi-i^ but n<i't v.i^>^the complitey fell be- % 
twe'en b(>th.otber gpoviSs in* achievement ai^jti ti-me^ 




Other resvilts bav^ ngt ^u^ed contrt)! g[mu4?;4e- * 
s i gns but. ha,v ^ u^e^d . p r ^ - pos t ' t't s t c omi>3 r is o]tV5 > .all ^ 
of ^ich sbtjw leatniTig was ^/o^npiishe<J/lflTH 'the - 
C.A.r. modules. " ; ' ]' V ^ ' . ' ' rj'. - ^ 

CONCLUSION i FUTURE PLANS '. • S*', i ' ' 

' — : .}.lt * ^^ % 

The Univers^t^' of Iowa' College of EdvcaVion Com- 
puter 3ased Education project has had a s.i-^i^if ic^'nt 
impact on *bofch student users ^and facult.y c^evfi^op- 
ers. The project was the result of a 'cooper4t ive . 
ef forKi^hictr included the University admlnist^A- - 
rion, fjT'r>"^B^^^^Ttilnisrr atiofi^ C^]^^^^, fa;f ' 



puter Center^ Computer Lab^ st«tf f , dnd* stiidenC^ H-n ^ 
the. College. ' , / \ <*• ' 

This integration of humaft haitlware t^cVinology 
has mgde possible instruc'tiona'l d^elopbent ^wbrlcfi^ 
otherwise would not exis< J One- result, -pr' thii^ in-. 
Cegration has-been- to provide a'rtev mode o( I^t 
sttuction to students in the teachejr^ education^prp- 
graras at The University of Iowa, i^ethaps the most 
exctting results have been wh'are Learning iJas irorc * 
-effectively achiex'ed through' X. A. I than through' 
traditional , -classroom methods. 

future pl^ans include commttnients to: • "^-^^ • 

1) Contlrtulng support for facUll'v development, 
^) Special workshops to train Vitcreta^i.^l and ^ 
part-time staff in . IDF and user assiitanbe, 

3) A" program of stude^st or^entatl^ln to ttic lab. 
and its resources is being revised,' 

4) Distrlbutioa. of products through -eo.NDU IT ia. 
being negotiated and,, 

5) Continuing evaluation of project** results. . / 
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THE TAKGLED TRIANGLE; 
• [* COOPERATION. 

COMPUTER BASED EDUCATION AND • 
PERSONALIZED SYSTEMS OF INSTRUCTION 



Kenneth L. Modesitt 
Mathematical Sciences Department 
Indiana University-Purdue University 
Fort Wayne, Indiana, 46^05 



ABSTRACT. A connsunity of knowledgeable adults is a desirable and attainable goal of education. Tlte 
concepts cf conununity. l^bvledge, and adult are discussed, as well as how thay are relatedTto the 
wurrespunJing processes of cooperation, computer based education, and personalized systems of instruc- 
tion. Looperation is suggested as a valua}>le aid in preventing the interparsonal isolation whlph may 
result from exclusive reliance on individualization. However , ^models for cooperation in a contesit- 
ba&ed education are sparse.. Computer science courses will be the content area* from* which si^ecific 
examples are d^awn. 
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Introduction • 

A community Qf knowledgeable adult? is a 
desirable and attainable goal of 'education. 
This paper will diicoss* the t)jree conccpts*of 
consaunity, kacwledgo, and aduXf; at\d how they 
are related tc Che corresponding processes of 
cooperai:ion, c ooputer-based education, and 
personalized j ys terns of i^isxruction, Ve 
will also cxp ore methods whereby cobpera- 
.tion can aid In bridging the isolation gulf 
which is a serious side-effect of many com- i 

, puter-based education systems and por^nalized 
systems of inistruction. Computer science 
courses will be ^e content area from 

, which .specific examples are drawn.. . 

^ Motivation 

I / 

Historically, formal education owes a 
great ddal to religion* .However, in recent 
centuries, much of education has ' intention- ^ 
ally minimized any reference to a god, spirit, 
sout,' etc. Unfortunately, in so doing, we 
are nov left with. a state of formal education 
which is almost solely derebral (preferably 
left-si^ed) in nature. Whereas, in the past, 
slavish attention was usually given to^ reli- 
gious concerns, now most educators concentrate 
•on intellectual needs of students almost exclu- 
sively. This is paijticularly true in higher 
education, although /the fact is certainly 
evident in lower grides. Now, inxollectual 
growth is obviously/ a very desirable quality, 
but, in my opinions it is most certainly not 
enough. I believe (that a community of knowT 
ledgeable responsible persons actively work- 
ing on a mutual, pnd represents an fnValuable 
goal for education. 

Computer-based education (CBE) and per- 
sonalized systems of instruction (PSI) are 
both remarkably, successful tools for instruc- 
tional delivery in content-based courses. 1 
have utilized both in a variety of computer 
scidnce courses over the past several years. 
Each tool will be discussed more fully in 
later sections. HoWver, CBE and PSI are 
both primarily designed to minister to the 
intellect of the studfent. That is appro- 
priate within the university, but it is not 
sufficient., 

Both CBE and PSI are addreised to the 
cl^ncept of individualization. Usually this 
means mastery-levtl lej^rnlng for each indi- 
vidual, often carried out in a self-pacing 
manner. Thfs process, I firmly baliove, 
is a far better al'tern^tive than an entire • 
term of the "me talk-yju listen (and learn?)*" 
professorial syndrome. The lecture as a 
tool 'for .content' delivery *nd acquisi'tion 
dents ij hardly adequate to thi 



by ^11 sAiden 
task 

F6i/all 



the many real advantages of 
bot^ CBE and PSI however, they share a com- 
mon side-effect I believe undesirable. They 
both prtomoto the likelihood of an individual 
working \ln isolation. If one believes, as 
I do, th^ a community of responsible infoirmod 
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persons is Important both to each individual and 
to the*solution of many serious current ^problems, 
then such isolation should not be an excXusdve 
mode of education. ^ * 

Cooperation via working on a common goal re- 
presents a positive resource in combatting the 
possibiiity of it>dividual isolation resulting 
from CBE and PSI frameworks. Meaningful joint 
efforts provide an arena in which the 4>er8on can 
exercise her knowledge, perhaps gained in a CBE. 
environment, w^th a responsible attitude, perhaps 
.gained in 'a PSI environment*. The Individual's 
intellect a^ maturity provide a foundation for 
significant individual and mutual action. Each 
of these vltal-edticational ele&ents: .intellect, 
maturity, and community, will be discussed in 
the' following sections as they relate to CBE, 
PSI, and cooperation, respectively. 

Coraputer-Based Education * 

Computer-based education (CBE), al?so known 
as computer-assisted instruction (CAI) , has a 
firm foothold in higher education today. I be- 
lieve that its greatest potential lies in an 
increase in the quality of cognitive education 
(difficult to measure) and s§condarily^ in cost 
reduction (easier to measure) [3]. However, 
money being what it is, the second benof-it is * 
receiving more attention, much of it negative. , 
Stories abound of how CBE System X was suppose 
to cost N cents/ student hour, and really costs 
2*N or 10*N cents/student hour. The latter 
figures usually incorporate research and develop- 
ment costs; tha former is usually one meant for 
production systems. The forthcoming evaluations 
by Educational Testing Service for the largo. 
PLATO and TICCIT CBE systems^ should help miti- 
gate the altercations [10]. In any case; it is 
becoming less an act of faith and more a fact 
of education, nhat CBE systems are increasingly 
widespread. Future generatiohs ■ of students will 
be hard-pressed to remain uneiposed to computers 
in their courses. *_I belioVe mils is primarily 
good because it introduces than to one of the 
most remarkable inventions ever known, as weljl 
as permitting them to learn many subjects by \ 
' doing rather than "listening, and often at their 
ovn pace. The increased quality of cpgnitivJ 
skills is the desired and, I believe^* attainable 



goal. 



— itv- trhe past , CBE has been considered the 
domain of computer science departments, with a 
smattering of tnathematici, engineering, and 
business courses also involved. I believe this 
attitude held, until recently, primarily because 
C3E was equated with haying students write pro- 
grams. However, this use of computers will pro- ' 
bably not involve f a large majority of students. 
This problem-solving mode is just one of several 
types of CBE. 

♦ CBE now includes the us*e of computers in 
education for: problem-solving, drill-and-prac- 
tice, tutorial^ simulation, games," inquiry, and 
Socrati^ or tutorial [18]. ♦ Xheae .divisions are 
not well-defined, i^nd several workers in the 
field have a different set of categories [A, 5). 
Kevertheloss, it is becomipg^clear that compiiterlfe 
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will Itnpact far more individuals in the '^student" 
node, i.je., taking a lesson, than will probably 
use the computer in an "author" mode, i.e. con- 
structit^g a lesson. The trite*analogy of "dri- 

, vers of cars" versus 'Guilders of cars" is not 

' far afield. , 

• ♦ > 

* It Us evident that a variety of organi- 
zatil^ns: governments, universities, and in- 
dustries, are convinced that the time is ripe 
fui introduciiTg an increasingly larger number 
o:\ students to computers. Funding of various 
^CBE efforts by the National Science Foundation 
Is. well known (10]. The PLATO nationwide net- 
t work cosponsorcd by the University of Illinois 
and Control Data Corporation has rfbcclved a 
great *deal of publicity (1, 8]. The intro- 
duction of the' CLASSIC* minicomputer system 
of Digital Equipment Corporation into school 
systems holds considerable'promise. And the 
availability of low-cost terminals is provid- 
ing pre-unlv^rsity school systems a realistic 
option ''or widesf>read computer use. 

Faced with the honestly exciting prospect 
, of "coming of age" (the first issue of a CBE 
journal will soon appear), CBE has not seriously 
investigated the more sobering aspect of pocen- 
tial^increased isolation of its users. Yet ^ the 
phenomenon is net ^nknovn. And^ as CbE cjcrntinues 
to be espoused as a tool for individualization, 
I believe care must be exercised lest education 
concentrate solely on the individual to thV 
detriment of the community. 

Personalized System ^^f Instruction 

PSI, al^o known as the Keller Plan, is one 
of the most promising approaches to content- 
education now' aVailabl.c. The literature 
is ample and growing rapidly [6, 15, 16]. 
journal will soon appear, and the national 
ferences are attended by. a wide diversity 
cators intensely involved with the learn- 
ing process. An informal and informative news- 
letter is available from the Center for PSI [14]. 

, Briefly V PSI ^concentrates onmastery-based 
learning » essentia], toj solf-resf}ect and profes- 
sional competence. In a, previous paper, I 
discussed how the use of PSI requires the inte- 
gration of freedom and responsibility [13]. An 
"aduft" was defined as one v/ho takes responsi- 
bility for , her Own actions, regardless of 
chronological age. 

' For those not faml^liar with PSI, the fol- 
lowing torse sunmiary should suffice.. It is 
excerpted from the paper referenced above. 
Briefly, PSI Is characterized by five attri- 
butes: 

a) mastery-based 

b) self-pacing * 

c) non-lecture 

d) ^written materials 
o)* peer tutors\ v • 

The argtanent in favorjof PSI runs-Aome- , 
thing lik^ ttfis. Students can gain in self- 
respect, and justifiably so, when they demon- 
strate thoy can master* the material (a). But 




I since students grasp material at widely varying 
rates, such mastery will occur only if the stu- 
dents are primarily responsible for pacing them- 
" ^selves (-b). Hot/ever, if they pace themselves, 
there is no way any lecturer could po8si4>le speak 
to all students at their individual points of pro- 
gress (c). But if the information is not tr^ns- 
. mitted verbally, how can it be -done? Written 
material provides a partial answer (d). With th(t . 
widely varying ratesi^jf^ progress through a course, 
there is no .way one instructor could mana'j^e the 
evaluation process for thirty studontjs working on 
twenty units of material, each with its own 
multi-version unit test. Hence, peer tutors be- 
come invaluable (e) . They each, including the 
instructor, take responsibility fdr about ten 
students. Th^ is, they go over the unit tests, 
answer questions, suggest resources, etc. 

As indicated, the desired course content is 
' splits into 15-2y)units, where each unit consists 



of: 



a) 
b) 
c) 



d) 



an introduction^ 
statement of' object ives 
suggested procedure for achieving the 
objectives 

supplemental discussion 
sample proiblcm/answers * / 

multi-version repeatable unit tests./ 
In previous papers [II, 12],, I discussed the com- 
ponents in more detail. Fo^ example, the objoc- 
tives are listed bv knowledge levels. And the 
suggested procedure involves any of the resources 
we've discussed earlier, t^g*, taking PLATO les- 
sons, in addition to readimg the toxt^ or con- 
structing structured* procedures . 

At this time, I have utilized a PSX frame- 
work six times, four of them in an Introduction 
to Algorithmic Processes course for freshmen % 
through graduate students. PSI has also been ' 
used in a dual-level Algorithmic Languagt&i course. 
Hesults are consistent with PSI studies reporteo 
elsewhere [7]: procrcistination is a problem^ 8d 
of the student* prefor| it to lecture, and tht 
same ones do well on final exams, retention, d^** 
^re,for another PSi course, etc. i In recent 
termsr] the peer tutors have bepn utilized more 
heavily by helping. to prepare clarifications to 
.unit ifaaterial and unit questions. The classes 
are not yet sufficiently Inrge to use. compliter- 
gonorated exams. Also, incompletes are not given 
unless the student has completed the number of 
units for a B. Completion rates during the se- 
cond trerm for incompletes given uncler looser 
criteria were extremely low. 

The maior cha^ige to the PSI framework is to 
int^roduco cooi3trat^on explicitly. I am,, of ^ / 
coCtrse, aware that ^ the instructor and peer tutors 
"cooptrai$> ," after a fashion, wtth the student to 
master tlve^course material. Y^t, individualiza- 
tion is one of the avoweJ astcts of PSI. As 
mentioned earlier, this mtode is a far better 
alternative to instructional delivery than the 



standard lecture 
hope for mitigat 



in my opinion. It offers real 
ng the debilitating atmosphere 



of competition bi sed on gradee, which are thedt- 
selvet often batid on speed. The ^0e of PSI to 
lessen tuch rivalry is one of its greatest , assets. 
However, at I hope to show in tl}Q next section, 
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replacing a, bompiet-ltlva-based classrooo' by one 
devdted exclusively to Che indivi!dupl is neither 
necessary nor desirable. 

Cooperation 

Webster's dictionary defines cooperate 
as "to act or operate jointly with another'^ 
[19). T he i dea of a common goal is also 
usually associated with cooperation. Now 
cocdon goals are hardly a new phenomenon, 
even when copmon goals such as cooperating 
to beat the other te^ are specifically ^ 
excluded. Cooperation to end wars, poverty, 
pollution, starva*tion and other critical 
problems is publicized daily. Cooperation 
withth a family unit to nurture oneself, perhaps 
a spouse, and possibly children is a good and 
often-realized ideal. And yet where does one 
find cooperation in the educational process — a 
process devoted to discovering more of oneself,, 
each other, what the world is like, and how to 
laake a v/aloed contribution? Why this dearth, 
when students enrolled in the same course 
S4PposcdIv have some, cortmoif goals? ^ 

Yet, looking abouC^'the world of education, 
there is little difficulty 'encountered in find- 
ing models of competition. What does Webster's 
have to say abouth competition? Competition is 
a "contest bcjtween rivals," where a rivfli i^ 
"ond of two or more striving to reach that which 
only one can possess." Sound familiar? Of cpurse 
it does at the Astrodome or Orange Bowl. But 
wha^t about the normal ^ell-shaped curve exhibited 
after so many hour exams, ffnals, and courses? 
What percentage of your students know 90-100^ 
OF the material y<fu expect ^of them?. Is that 
rarified level reserved .for only a few, say lOX 
of the class? Do you "rewarii" only the fastest 
<co master your course objectives? As indicated 
in the last section^ I believe one of ghe 
greatest assets of PSI is the lessening of 
such competition based on speed. 

. p^' 

^ Consequently, jfj PSI does in fact reduce the 
"what*'jde ^et" 'syndrome, what replaces theWacuum 
in the 2;ealm of ihterpersonal relationships? I 
belicv^ it is toovoften indifference. If e/ery 
student Is tutor coi as "a class of one," we are 
going to raise somk very socially-impoyerished 
students. The neeAand desirti .to relate to 
* other students couloy diminish drastically with 
increased individualization brought on by more 
widespread use of PSI and CBE.. 

I believe that the usual competition-based* 
class, as well as the completely individualized 
clas^ can be replaced by t'he class based on a 
knowledgeable (CBE), mature (PSI) cdmmuivity 
learners. The community arises troh coopera; 
tive efforts in a broad spectrum of sharing 
activities and thoughts. The following exam- 
ples of such cooperation are drawn from the 
introductory computer science course. 

Perhaps Jihe most major cooperative effort 
derives its genesis from an exercisja in the 
excellent reference. Values* Clarification [17). 
The exercise has the students form their own 
lists of "What I know about X, and what I 



want to know about X." ^X is computers in our case. 
From the latter lists, groups of 3-4 are formed 
based on a common interest. The assignment of 
each group is then to research their coirmon inter- 
est for presentation to the rest of the class at 
the end of the tJerm. * The presentation is not ' 
having someTone read a research paper. Rather, 
they are often multi-media in nature^ video 
tapes, interactive computer programs, stories, 
photographs, plays, music, etc. And the topics 
chosen cover a very broad range: med^ine, 
graphics, weather, mathematics, landscaping, 
traffic control, building a computer, consumer 
indexes, etc. As our campus happens to be non- 
residential, one class hour a week is devoted to 
"project day" for those who wish to work in pheir 
groups. Other students may wish to take or go 
over unit tests, study, or jcilarify a point with 
the instructor. T^jis is a aormal'mode in PSI 
cla-sses. Other cooperative efforts vary widely, 
depending upon the ingenuity of the instructor 
and the students. The Values Clarification re- 
ference mentioned earlier contains a wide variety 
of such exercises. The course includes a sizea- 
^^le number of films dealing with computer applica- 
tions, moral-ethical questions, and technological 
concerns of oomputers. A sample list is in 
Appendix A. Round-table discussion follows each 
film with all.peoplq sharing their values, if 
they wish. ^ 

Other cooperative efforts during class time 
include taking undocisnented prbcodures written by 
other students, and working together to analj^ze 
what problem i« being solved. Also, since 
dents do not have a »eii^e of competition during 
" the course, they of^r and accept help from other ^ 
students who may be ^ unit or so ahead at the ^ 
time. Such aid could be on content questions,' or 
on how to obtain a listing at a remote batch ter- 
minal by submitting a job from an interactive CRT 
terminal. One o'f the tDOst challenging cooperative 
efforts is for a small group to de*sign an inter- 
active computer gajjto based on cooperation-, rather 
thi^n comJ>^ti&*ion, e.g.. Star Trek, Moonwar, Battle- 
ship, etc. One Of their strongest responses is 
that they Ijave not had any models for games which 
are cooperative in nature. How many do you know?? 
This fact alone says ^ great deal about the value 
systems espoused today. The desirability and need 
for cooperative efforts rather than competitive ^ 
ones in the world and in education today, is ver^ 
great,' in m^ op^lnion.' 
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The students are hardly alone when they com- 
plaiV they have few models for designing a gamA 
base^jon cooperation. I^ have a difficult time 
discovering or creating methods to promote cooper- 
ation Within a course. The few mentioned previoua- 
ly are not. many, but theV represent a large in- 
crease over my original virtually null supply. 
The design of content-based courses to include 
cooperative inter^iersonal relationships is a far- 
tile research area, in my opinion. It does not 
appear, for example, in The Yellow Pages of Under- 
graduate Innovations [9] which is a handy r^f«r- 
enco for many other experimental appraoches to ' 
educalion. 
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The belief persists, nonetheless, that 
cooperetion withjLh the classrootn is not only 
highly desirable, but also possible. A re- 
cent articjje in Psychology Today I2] dealt 
with such concepts at elementary grade l^els. 
The article was written in the context surround- 
ing the forced busiisg for integrated^piucation 
Issup. References to additional sowces of 
information would be greatly appreciated. 

The introduction of PSI and CBE into the 
classroom has done a great deal to alter the con- 
cept^of a professor as the person standing at the 
front of the room dispensing information. 
There are many educators who are excited about 
forming a new -role'* fo^ themselves, but als*o 
many who are not so interested. The incor- 
poration of cooperative efforts to help build ' 
a community within the structure of a course 
will also change the role of .the instructor. I 
believe It is. too early to say how. 

Summary 

A commUnKiy of knowledgeable adults is a 
very desirable goal for education. Computer- 
based education can do a great deal to improve 
the quality of intellectual content (knowledge) 
gained by students. Personalized systems of in- 
iBtruction give the Student the opportunity to 
/take responsibility for her actions which in- 
volve considerable freedom (adult). Cooperat:^on 
based on coinnon goals is a substantial aid to 
bi^lding a community of students. Moreover, 
.Cooperation is not Just' a frill. I believe 
th^t cooperative efforts are necessary to pre- 
vent the increased illation of the student in- 
volved with the individualized instruction 
espoused In both CBE and PSI. Unfortunately, 
far too many of us, educators and students ^. 
alike, have *a multitude of models for com- 
petitive efforts and a dearth of models' for 
cooperative efforts. 1 sincerely hope this 
\ effort will prompt more interesd and work 
\ in the design of content-based courses which 
include cooperation as an essetitial component. 
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Appendix A 

Ken Modetitt ^ Fall, 1975^ 

Computer Science Film -Schedule 



Sept. 3 



Sep 



(HedO 
CWed.) 



Sept. 2A (Wed.) 



The Computer Revolution: Good 
overview of computer applications 
as of late 1960's. Part of 
Cronkhite's 2l8t Century Series. 

Computers rat Work : Good applica- 
tion Flick (cakes, toys, etc.) 
Comp.ut-Her Baby : Graphic^ spoof 
of computer dating. ' ^ 

Incredible Machine : Bell Tele- 
phone plug for computer music, 
graphics, motion pictures. 



Oct'. 1 (Wed . ) The Right of Privacy : Superb 

accotint of ways in whicl^insti- 
tutiqns invade the right^of 
individuals. And how the Com- 
puter can "helV". 

Computer Sketchpad : Good early,^ 
film of applications and methc}.44 
of computet graphics (CRT-based). 
Bart Automatic Train Control s 
Short ^animated promo 



Oct. 15 (Wed.) 



Oct. 22 (Wed.) 



Nov. t (Wed.) 



The L'eading Edge : Excellent in- 
depth coverage of computers in 
aircraft ma^^ufacturing : model- 
ling, pi;oduction control, man- 
agement systems. 

The Catalyst : Univac promotiorjal 
filfn on potpourri of applipatiops 
-somewhat sensational. 

Discovering Electronic Music : 
Grdat film for music lovers. 
Shows how synthesizers are 'con- 
structed and how computers aid 
composers. Far out sounds. 



Nov. 12 (W^d.) Thinking? ?Machine8 ; Good*- Bell 
Telephone Flick on different 
types of ^'thinking" and how well 
I * computers do with each type. \ 

^ Robots Ge^ Smarter : Late 60*^ 

• 'account of artificial intelli- 
gence work on hand-eye and mo- 
> bile robots. 



Nov. 26 '(W^d.) 



Dec. 3 ,(Wed.) 



A. 



Spires/Ballots: Review of 
work in information (document) 
retrieval in a cojnputer-based 
system at Stanfbrcl'. « 

Matte//of Survival : -'Scare^fUm 
of <hoW one nan ±9 affected by 
his' company becoming automated. 
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Dec. 10 •'(Wed.y CAI in^^ew York : The use of pro- 
^ . Co type computer -assisted ihstruc- 

,tloh (tAI) In. « New' York City 
elementary school. Quite a con- 
trast^ to standard modes as veil 
« / " as to PLATO efforts. 
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THE CALC'ULATOR : A BOON TO EDUCATION 
* * 

Pearl A. Skoll 

Proje.ct bir.ector , "The Mobile Computer Math Lab " 

Los Angeles Unffied Scho^ol District ^ Area K 

8111 Calhoun Ave., Panor ann a City, Ca^ 91402. » 



ABS TR AC T : The impact of the computer revolution has been upon us for several decades and will 
continue to aflPect our lives for generations to come. Miniaturization of el^ctroaic components has been^ 
responsible for the devlopnrient of the minicalculator . 7*he economics of our country has made this a 
product available to the consumer at a very reasonable price. Thus the minicalculator has becon;ie a 
responsibility of the educator. It is necessary that we learn to cope with the calculator and create ways 
to use this highly motivational device cis an instruct lonar tool in the cleissroom. Documentation of the 
success of students in mathematical achievement was nnade possible under; grant awards ffom Title III, 
bi^EA, from, 1971-1974, fqr the ''Mobile Computer Mathematics Laboratory". The project developed for 
grades 5-12 m five schools of Area K, LAUSD, has now expanded to 40 schools, grades K-12. Students 
^vse a variety of calculators, from desk-top progrannmables to the hand-held minicalculator to learn the 
concepts of mathematics and to improve basic computational skills and understanding of numbers. One 
vital component of a calculator program is the traiOin^Of teachers in proper techniques for using mini-, 
calculators in the mathematics class. Administrators, teachers, parents and students are involved irt 
setting local policy regarding the utilization of calculators in math classes, within th^*framework of the 
mathenfetics program . Thus , through creative teaching strategies , and students* proper use of the 
calculator r — the calculator is, indeed, a iDOOn to education ! 



Yhere is no doubt that the computer will continue 
to affect society and the world in which we livfe. 
Due to technological advances and miniaturization 
of electronic conr^ponents, the size of computing 
equipment has become^ so relatively small that* 
calculators with a great deal of mathematical 
skill fit in the palm of your hand. 

,-/. i 

Industry his developed a product that is available 
to the general public. I refer, of course, tq the 
minicalculator. Due to the economic situation in 
our country, the calculator has become a "thing" 
of interest, excitement, and pleasure to individ- 
uals of all ages in all walks of life. When a calcu- 
lator i^^sed correctly, a problem is solved accu- 
rately, in a minimum amount of time. Thus the 
drudgery of laborious calculations does not stand 
in the way of understanding, and learning develops . 
The emotional feelir\gs one derives from "success" 
cannot bj^^ritten dcwn - for there are varying 



degrees of reactions. ^ 

Thus J it IS the responsibility of Educators to offer 
"success" to thevr. students in a ma><imum number 
of learning situations in a givert time period. A 
studertt's "curriculum" refers to everything that 
affects his daily life. Whether one names a 
subject area "science" , "home economies'! , 
"driver education" music" , "art" , or whatever;; 
when numbers, size and shape, measurement, 
relationships, and logical analysis of a problem 
require an accurate numerical solution, it is the 
arithmetic of mather^aatical procedures that does 
determine whetheV this answer has been calcula- ^ 
ted correctly. 

The minicalculator has been built to do this. 
Correct answers "pop out" on a display screen. 
Fantastic I? Not really! This is true only if an 
individual enters the correct numbers\ in a correct 
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procedure , within the capabilities and. capacity 
dfthe particular calculator. , 

Now a variety of problems and^concerns face 
administrators, teachers, parents and stud- 
ents. The calculator - a boon to education - 
must not be misused; for abuse of the calcu- ' 
lator IS more detrimental to education than not 
using It at all. The power of this tool must be 
contained within the framework of a meaningful 
instructional program, that w lU continue to 
offer students the opportunity td improve their 
basic conrfputational skills while learning and 
understanding the concepts of mathematics. 

Thus it IS the responsibility of the educator to 
expose students 4:0 the various types of calcu- 
lators in meaningful ways. One mu^t devise , 
create, cope with, or develop situations of the 
world in which we live, to which the calculator 
can be adcip^d^". — 

The calculator is a. highly motivational device^ 
Everyone wants to at least touch one. ThiTs the 
educator must channel this enthusiasm toward 
learning experiences. Then the calculator will - 
take Its place in the classroom, along with paper 
and pencil, chalkboard, books, and multi-media 
materials and equipment, to fulfill its rol^ as 
ant instructional aid and learning tool. ] 

Similar to the developmental processes required 
for a baby to proceed from cra^lxifig to w'alking 
and running, are the levels of abilities of students 
to work with and use calculators^ as they pro- 
gress up the ladder of learnihg ^and understanding. 
Although a calculitor will compute an answer , 
it will be the decision of the s^tudent whether to 
accept this value or not, sm a reasonable answer 
to the problem.' Thus students develop a "number 
awareness" through the ability to estimate answers 
andiconsequently accept or reject the. figure that 
* appears on the display. 

•Students become interested and involved in their 
mathematical studies. Enthusiasm is generated 
to explore, think, deduce, predict, ^d»solve , 
problems - for the fear of mathematics and the 
failure previously felt because one does not know 
the entire "multiplication bable" ,(as one of many 
-illustrations), no longer stands in the way of 
understanding.* The calculator^ fills the gap be- 
tween understanding what to do to solve a problem, 
and the actual computation of the numerical answer. 

The creative, cistute educator will find that the 
calculator may be used.a3 an aid in improving 
perceptual skitls-of the psycho-motor .domain; and 
behavioral attitudes of the affective domain; all 
this simultaneously during the period Of mathe- , 
matical growth. With reference to perceptual 
skills, educators abree that some mathematical 
errors might appe^ $>^cause a youngster has k 



A. ' ■ - 

Vision proble'rr^\ or reverses numbers, or has 
poor handtA/rlting, or poor;rnotor development, 
or many other types of problems that have 
nothing to d^fwith the rules of arithmetic. 
Through a variety of activities on the mini- 
calculator, many perceptual problems can be 
corrected - mathematical skills eventually 
improved. 

Educators also s^gree that leeu^ning is diff.icult " 
when poor behavioral pattern^ and attitudes exist 
within the educational environment^. Since we* '* 
find, that students are anxious to use calculators^ 
for'^Che study of mathematics,' what simpler way 
is there to develop self-djscipUne within a child 
than to take a calculator away and c(eprive him 
or her of using one until one can show self-control* 
and respect for property? Once again the calcu- 
lator has come to the aid of the educator lO the 
growth and development of the child. Now the 
students wants to learn, and so he will — and * 
mathematical skills are again eventually impro- 
ved. 

// . * * 

O^'not take this statement hghtly ! It is not mten- 

^d to mean that the "magic" of the calculator 
always works wonde^, or ev^n ver^ quickly ;* \ 

.but rather that it possibly can do something where 
other efforts ha\^ failed. 

This year, we are finding that our work with 
kindergarten youngsters lis Correcting recognition 
difficulties. Children had been reading the num- 
eral "2" as a "5" , or vice- versa; and eyen^read 
the number "5" as the letter . Mff^e con- » 
vinced that th^ earlier these types of errors are 
corrected, or" at least made "cognitive" , th^n , 
they can be corrected and no longer cause the j 
child to*^^e>?perience failure in school. 

Many of our students fipd that th^y can compute 
dertain problems faster thah the. calculator. The 
challenge then becomes one of using larger num- 
bers, or problems of higher dlff.iculty. Students 
begin to understand the "how" and "whv/" of 
mathematical computation through reasoning , 
WatKet** than by rote memorization .of a technique 
Id to thenrt by a teacher. 



The low and underachieving students ar^ encou- 
raged to study arithmetic because they are, now 
st^ccessful ahd understand the "bnderlying concepts. 
The gifted students are challenged'to higher 
levels of analysis in their understanding of 
nr^athematics. 

/ • . ^ 

An underlying ingredient for the Acceptance of 
the minice^lculator by society is the fact that it 
is 'Simple. to operate. Anyone can push the keys 
and have numbers appear on Jthe display. 

Now the^c'OntWversy is heard ! "Children -are 
cheating when they use a. calculator." .... 
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"lt*s a great gimmick." . . /'The calculator is^* 
SQ cheap ncv/ we can txiy a class set ." . . ."^h^ 
should the schools buy calculators if everyone 
IS buying his/her own?" . . ."When you learn the^ 
basic facts, then you may use the calculator in 
school.". , /'If nry child uses a calculator nov 
he won't be able to balance his checkbook." . . . 
"Do'you lillow the use of calculator^ during a 
test"^" . . ."Don't you ever bring a calculator into 
my classroom!". . ."Show nr>e what this gadget 
ca^ do, and then maybe UH use it fn my clciss." . 
. ."vVhat security mesisures ^can we take to prevent 
theft and vandalism?". . ." l%,this calculator V 
elec^^^ally safe for the children ?" . . .etc. etc. 



These are but some of the concerns to society. 
Educators cJ^oss the-nation are attempting to 
deal "With the "populan^" of the calculator. 
Its lOipact and existence cannot be ignored. 
There is no doubt that the calculator has a role 
to perforrr withm an educational progreinn , The 
calculator in itself is not a panacea. Staff 
development is an important component of a 
caUulator program. ' - 

A teacher should feel "copri for table" in how to 
use a calculatoj to achfwve definite goals and 
objectives*. . . for, no matter How "cheap " a 
calculator)(i3 it i^ too expensive to be used 
solely cis a'check to see if calcC)lat\ng wcis done 
correctly t>y paper and pencil. An answer 
sheet servps this purpose very well. 



CXjr program, "The Mobile Computer Math Lab 
funded from 1971-197^1 under Title III, started in 
five schools in the^ San Fernando Valley (grades 
D-I2)j There were twenty classes involved m • 
the program. 

Test scores over a three-yeeir ^period, 1969-1971 , 
indicated a need for concentrated mathematical 
instruction. In^two of the five schools targeted. ' 
to receive services under the project, test scores 
were *^<i and ^i percentile points lower for 6th and 
12th grades, respectively, than the average of all 
Schools in the district. 

A van wcis purc^1^lscd and converted into a mobile 
mathenr^Qt ICS center. Equipment and educational 
materials were rotated amongst the schools as 
the projiect director traveled to each schocj. A 
variety of calculators included desk-top program- 
mables, memory-type calculators, and those that 
used either paper-tape or hSd a nixi-display screen. 
An x~y*plottor wcis al^o used with the progreimmable 
J calculatoi^^^Ouipment was lirnited, so a variety 
of hlmstrips < slides < cassettes , math kit^ and 
geuTjes were also used to sc/pplement and enr tch 
the mathematics programs at the project Softools . 
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ittaff development and community involvec^^'ent were 
components vital to the success of the project. Once 
the teachers^ and students learned thg real value of 
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proper use of the calculators in the teaching and 
learning of mathematics, their enthusiasm for this 
type of learning situation soared. 

The final report in June 1974 contains the fqVlo/v- 

ing in^^mation on effectiveness: 

" 61 .6 % of all students who took both the 
pre- and post-test were enrolled in 
grades in,which statistically significant 
progress wcis made, (\n terms of pre- 
post "2" scores of the applicable *norm 
group) i 1 1 .9 % of students were on , 
grade levels in which no statistically 
dagnificant phogress wcis nnade; and 
6.5 % of students were in a grade level 
in which progress Wcis significantly ^ 
' negative." 

In addition, this final report includes some very 
important, though unanticipated findings . These 
^re: 

^ Xa) Student attendzince improved. 

(b) Parent interest increased. 

(c) Teachers at the project schools used 
techniques they learned in demon-' 
strations with students who were not 
officially included in the program ^ 
and/or evaluation design. 

(6) Teachers in the project schools who ^ 
were not included in the demonstra- 
tions took the time to learn our 
techniques and used them with their 
students . 

(e) Teachers in schools, not included in 
^the project, but; who had similar types 
of equipment, picked ^ the techniques' 
used in this program. 

(f) Articulation heightened amongst teach- 
ers of various grade levels from the 
elementciry and Secondary schools . " 

A'lth the decrease in the cost of the minicalculat©r , 
many schools began to purchase their own types 
o(^ equipment- Requests for proper techniques 
for using calculators to teach the concepts of 
mathematics were received from n^any schools 
in LAUSD. Title Ml funding was no' longer; 
available — yet Area K decided to continue** 
this project. For the schopl year 1974-l97b,J 
our services were expandep to include 23 schoqls. 
The 1975-1976 school yec^|shows further e>|pan- 
sion. J .^^TPQj^^e now travel' to 40 schools in Area K. 
Occasipnal demonstrations are held in other* Areas 
of LALSD, as requested- (Ther^ are 12 A^eas 
in LAUSD-) Our expertisi^ is also requested 
$tjprOfessional /nathematics conferences across 
tP>e nation, . - and as an extension instructor at 
UCLA, CSUN, CSU-LA, P^ppter dine University 
and a Community Adult School . 

Thus, w^eel that the calculator i^'not a gimmick, 
nor a crutch". - . bgt rather a valuable , exciting 
tool that, when used properly and wisely, can help 
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children to learn- what mathennatics nsally is ! 

Further conclusions discov^ed as the nninical- 
culatpr continues to invade our classrooms are: 
" \^) Calculators take the drudgery out of 
• computing. 

(2^ Calculators cannot "th)nk" , and con- 
sequently do only what they are told to 
► do — thus, students havd to Know their 

mathematics, and press the right keys 
*^ on the machine to get a correct answer. 

(,3) Much more number awareness and 
critical thinking are required when 
using a calculator correctly and effi- 
* ciently for the" study and learning of 

mathematical concepts. 
(4) The lew and underachieving students 
are encouraged to study arithmetic 
because of the success factor m their 
^ learning of concepts. The gifted 

\ students are challenged~i:o higher 
levels of analysis in their under- 
standing of mathematics. 
(5^ Students have a great respect for the 
calculator's efficiency and capabili- 
- ' ties-. Enthusiasm for mathematics 

ranks very high in a class where 
calculators are being used. " 



-Most teachers take prtde in their profess ^bo and 
>*ec?eive nnany rewards through the s'uccess- of 
their students. 

The calculator has the potential ^ help "turn 
youngsters ort " to education. . . in this case, 
through^the learning and understanding 
mathematics . , 

It IS, therefore, the 'responsibility of each local 
school's administration, teachers, parents and 
students to discuss the implications of the 
existence of the calculator; and to develop a 
poUby concerning the utilization .of calculators 
within their educationaj prograhn. 

Through creative teaching strategies, and 

students' proper use of the calculator 

the calculator is, indeed, a boon to education ! ! 
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FACULTY RATINi PCTLICIES OF COMMUNITY 
COLLEGE MATHEMATICS S7UDEI:TS 
Verrxon M/ Guymon or. 
£cottsdale Community College, Scottsdale, krizoua 85251 



Tne Purpose of tne study was to 'determine what faculty rating policy or poll- 
:ei z'dbsets of MCCCD m§t'hema,tlcs students used during the 1975-1976 academic 



of thic study was ^o deter- 
raring policy or policies 



AMI fa: Jit: 

't-.i r.L^?et3 of Maricopa County Commu- 
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District (MCCCD) students used 
-.fT tr l/:'5-1976 academic year. ^ 
F-^oulty' evaluation by students is an 
j^reat <^oncern and the focus of much 
jc'i* If nal research. In 1^7^, the MCCCD 
rcver'^.lnr coari dire:;ted the distri-ct fa- 
:^lty to develop a faculty evaluation pro-. 

wr.ich would include student evalua- 
tlc-"*' ::r faculty. A faculty committee was 
cr-'^.n'lwMon-'^d to create a proposal to be 
submitted f?r district fa6ulty approval. 
In Decenc-r, 1975, the committee's proposal 
war presented and voted upon oh a campus by 
campus-^tasis. Two of the 5 district cam- 
r i3e3 'approved the proposal. Each campus 
^ Which rejected the rror'osal was given until 
Marcn, 1^77, to develop an evaluation pro- 
cedare ^;itr. the stlpulati:>n tnat the proce- 
dure ln:lude student evaluations of faculty. 
.■:t..dent evaluations of teaching effec- 
.^73 r.ave been found fx be related in 
• -^rnr. student rating po*licie?. Identi- 
fying^tnese rating policies has been a pro- 
tlen. However, a technique for capturing 
ar.i clustering raters' policies. Judgement 
An'ily/l^ U^AN*y, wa? suggested by Bottenberg 
and '^hrl^tal (1) and Christal (3). In a 
;^tudy at the University of Northern Colo- 
rado, Houstan (5) us»d JAN as a vehicle 
for identifyinf: different Judgmental poli- 
cies used by various subsets of student 
raters. Regardl'^ss of the grouping of 
students (grade level, reasons for taking 
the course, scbiool or college in which the 
^students ^ere enrolled), one faculty ra- 
ting' policy emerged for the student groups. 
The polic^^ was described as' student con- 
cern for ^|he personal characteristics of 
^ the instri otor (instructor's interest and 
"enthusiasr, inter-oersonal relationships, 
ability tc communicate the subject, abili- 
ty to comranicate tne subject, ability to 
Interest ?nd motivatV students). The in- ♦ 
r-tructor' s 'classroom yn^nagement charac- 
teristics and professional qualities (at- 
tltule, kr^^wledge , and preparation) did 
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not make a significant contribution to the 
students ' policy . 

Following the procedures used in the 
Houstan study (5), JAN was used in a pilot 
study to identify different Judgmental po- 
licies used by varioys groups of mathema- 
tics students attending Glendale Communi- * 
ty College ^nd Scottsdale Community Col- 
lege during the 197^-1975 academic year. 
When students were grouped by the campus 
which they attended, one faculty rating 
policy emerged. However, when ^he stu- 
dents were groupeJ by their mathematics 
instructors twc% faculty rating policies 
were obtained. 

PURPOSE OF THE STUDY; ' Tb^e purpose of the 
study was to-,determin« what faculty rating 
policy or policies sel^at^d subsets of 
MCCCD mathematics stud^^nts' used during the 
•1975-1976 academic year. Three grouping 
methods were used. ^ Specifically > students 
were grouped by*campuse3, mathematics^ 
courses-, and by mathematics' instructors . 
By the use of JA^, the identification oC 
the minimum number of different Judgmental 
policies was made for each of th^ studejjt^ 
groupings. Student evaluations of faculty 
were conducted in December, 1975, and in 
May 1976, .to determine if the faculty ra- 
ting policy or policies of the various 
student groupings varied' from semester to * 
semester. 

HYPOT^g^ES: The following hypotheses were, 
•testeorin this study: 

HI. A Single faculty rating policy 
is ob^tdined when students are grouped by 
campuses in the December evaluation. 

H2, A single faculty bating policy 
is obtained when students are grouped oy 
mathematics courses in the December eval- 
uation. 

H3. A single facufty rating policy 
is obtained when students are grouped'by 
mathematics instrutors in the December 
evaluation. ^ ^ 

H^. A single faculty rating policy 
is obtained when studeAts are grouped by 
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' !• \* ^ '>.'il.j"l •. :atiell': c '^r^'^e^ric notation ac:> fcllCwG: 
c/*;, /i-rl- f'l: .1^ / • j^:!:/ - V livariate. Multiple mzvez- 

i-^:: A-.er. stud-e^: -"^ /r^'^'-i r rlcn ctaiier ar^r act-ually tr.e rr.iaroir:t ber 

c.Tr:-/ i'. ' <^l~'\^ twe^r. civ^ria"'^ and ^multivariate work r^ut 

« ^i'i^^le fa?-.!:- :--iMr*- r ^r^^ considered as' tivariate for jlascifi- 

.-rd .<r.vn ::^i-r.^ : •» i' ^ : - -arirn r^rr-rcc. 

^- ir,/tr.:"'r V *• '^i ' ' i' - frr "f'^eeiy happening.** The 

variatlej ^.ea.:ured in this study were not 

n rt-i-i."* '-^A^ \ ''jiriipulated -MCr interfered with. 

'VJ .It • • * 1 - "^atedc" Variable 1 was ^.ea- 

\ • i.y zzai * ' • - s .*ured : ef rr- variable 2, variable'? wa3 

■'.1/ 'ar--^") t . V • * ' --easurc-d cefcre variable 3) etc. 

rirll?v 0" : " -• T'^^r "uncontrollea. " All of trie 

1-: \' xi Xav o*-al.'i"^ • . » ror^sioly involved variables left unmeasured 

a-^> ' -f- and unobserved (classroom facilities, time 

V. : :f day cf ^la^ces, etc.) were allowed to 

^.*-:} I; % . ' /. differ ao tney will. 

^^^c.^ -r«,.. . • • ^ 3 f^r "aostractive . " Tne research- 

: • . _r .^"i ^^y •:'r cr.^'r-^ v.'^i?:* variat^lec to observe accord- 

,'\-^' ' Inr to a cr^clfic hypothesis about rela- 

• .-^ ► • : ' Morrt* a'-or.r a eet of variables of interest. 

*: r .0" r . ' ' • ' ^ . • • ! r / - '^^r "reved to a oiased sample." 

<^ u '\ " '- ' • ^ " • Faculr^ -^e-^ters wno volunteered to parti- 

' ' ' i^.-' ► J- . - '/.'v ^ipate in t^ 1.: stvidv were selective in de- 

-r : :1 *• . " ' '"'iv '.ermininr .-.'^icn classes evaluated tneir 

. *• f" >** n 1^ -> <a 

^ : • « II.'rTRUHE::!;^: I'JiJ: Ctudents were asked 'to 

- .-^ ' MnK tneir rathe^/itics instructors on eacn 

il.-^lr.i:. ' :1: : • ite^ of tne^followinf: instrument. 

V: /.i- Hv.*i'j;.:io:: '"f i::rT^urTior: 



'\ ' / ' . ' 7'eare rate only tnis instructor in tnis 

' ' ' 1 r articular course in accordance wicn this 

' * rat i'lr^ r.cale : 

r-"r (^) r^ir f 3') Average' ( ^) 'lood 

* rt," * * \. - 

] ' \' \ : — rr.--r'/ i:lrrr.^t a:.d 12 3^5 

* ^\ ' * ' ' " " e:.tn 1 1:-" f'^r --.irr*^-. 

^ « • . ^ • P. ;:ilitv i'.- iuatelv an- 1 : 3 ^ 5 

_ \ ^ ^ .... '^biiit.' to ^j^nrunicate the 12 3^5 

. .-s,^ : y'xy^:. . ^ ni'tr-r -ffectiveiy. 

^ . , . ^ ^^>^^ Ability to Ir.^orerr and no- 12 3^5 

*^ " ' ^'^''^ ^ Pat-rner.^. in t- ^ Inr? and 12 3^5 



'r 1 



1 V " 1 *' 



.--"ar; ^f ; ' 
f 1 : 1 * V 1 ' 



"-i'l'ri to /t.it^n^^r needs. 

^ ' ivr^ Tt Ivl / »ire , 1 s ^ 5 

"..urr.t> ."b.^ ---^t V;ec are met. 12 3^5 

^•V'^i'\f,inr ocn.-idored, in- 1 2 ^ 5 
ol^di'u- rtrtnrtn.-. and weak- 

*"rr^*^'.\, ""I w'uld rate th*^ 
* '^ac-f.er . 



- \ .t .iv ... ^ , • •t\ 



ilV ir.'-'r.nrnt wac used by Houston(5) 
->rr ino ^-.Kit r^licief^ existed for var- 
i.1 f^j:^ :f .'t.-iderit raters at the Uni- 

■-rritv <^f^ Nrrt '>t'!'n .\^^lorado., Tince 'the 



*' "^'^*', ^ ' 'i^ \ in.-tri.ront i.> iv^t widely used or recoF;nl3ed, 

•J • -r'l^^J i ; . "V ^- -r-v 1r^r.->-n>.t t lo m.^ 1 de r Its rell-~ 
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rrv irr^'':**ant to Iccnsider Its reli- 
-^^liabiUtv - reliability of an 
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inctrLiment refers to the •stability , re- 
reatacility, or the precision 'of the d^ta * 
coi;«?cted fron tne instrument. The re* 
*'r^n:eJ or. the first 8 iter.s on the in- * 
-^*Turer|t were considered as Dr^di-ctor vari- 
at'i.*:^^ Trr tr.e response on iterr. ^* In the 
•i-irt?^ iiy, the (naltirl-*R ?.quared)^ 
"aljes excf;eded .7^. This indicated a ' ' ' 
^'tirh lerrfe of intrarel iabl litv for- the* 
''.^rtr.t-r'- ?::ora1o ctudentc. V.'hen data 
A''=^r*i' ^-^olle^te I ustn?: tnis inntr'jment at/ 
T'^^^liA ^-"^ the pilot stugy durirl;^he 
XQ":.-.'*. 0 7^;, ,'-r,ool year, the R*^ values ex- 
re- a*e J ^Theje results indicate are* 
"lae N.-.-c^- l-.T'ee of rellaciiit'; for the 

V-^xi ii; /> - The validitv an instru-. • 
rt-.- r-^-^T'^r/ to tr.e extent r wr.ich data 

"'.^ 1' .^"rurient neasure .-/-iat they w^re 
*-*-nl--. 1 * . '"oYr.are. Any -wtimite of the 
-X*.-:/ - '.'1 h instru?:ent ir. val-M re- 
'..r-"*' alroadv exist inr i.-iStrur.ent or 
''•""ij >*re nac Safficient relevance, 

1 lir 1 1 ! / 'in 1 validity to ce ^fed as a 
:;rt"crior. '^o'^irure.. In the donain of tea- 

r.v> ^.^t>^^^i^,,^^^^.: criterion is not 

^vallac'l-^^. .••^A'ever, "rats f^) used a 
^.I ire-" T .o. *■ i or.naire to determine the nun- 
c^r and ir--^ :r th-^ ra:torr whicn under- 
li«^ ^de^t rerceptions of teacners. A 
'I'.rle fact'r ac^^unted for 1.5- of the 

'ir*'in:e in test iter.s. Tr^ rirnest load-i 
i-..-- -f***^ic f'lctor vere (a) "clarity \ 
-v: i-i^riti-r./", ft^ "fairness", (c) "at-/ 
"lt.,1- to.:a^d rta.ients", (d) ".St ir.ulates 
i^t-^r-^s" " , fe"* "attitude toward subject", 

f "attl" jle t:'.vard student opinions\ (g) 
'•--n:.- irar-'-e'-t of student participation". 
Itt**s 1 trr:'':f;h or the instrument used 

tr.is ctu«uv are addressed to these con- 
cert/. /Iters and 8, while apparently not 

\rv:l'^iz related to student percep-- 

M "^n ' '.eiO':ers, .v^re retained for com- 
^irio->n '.%iti. the Housto*: st-udy. 

l/iT-; "GlLECTIO)i' : A prepare i packet of 
^ i-^st ionnaires and answer sheets was given 
"i^h T'ar'' I cipating faculty mer.ber'for i 
111/ classes. Tno packets wer^ *( 
ilstritjt-^d to the f^K^uU^ during the\weeks 
tof :40v^rter ?^th and April l?th.^ All pac- 
,i:eix> Aere collected prior to finals at'' the 
iend f oa."'h senerter. ^r. the frint of each 
ipi^r.'j*- .vas a cole for th" specific class 
iand !i.str'i>tor and a set of instructions 
rep'ardin^ the adninisterin?: of the ques-» 
tionnaire* Briefly, the teaciier was in- 
structed to chos"e a student in the room 
,to adr.inrt^^r the survey and then leave the 
ro7^ whllt^ the survey was ^onoleted. The 
packet was then delivered by the student 
'in charge to the mathematics office. 

Jr<ri\TMK?JT vO? THE DATA: The students were 
^v^\it-^\ hit,' three selected subsets. The 
first grouping was made by campuses with- 
*ln the MC-'^cr and 'resulted in subsets or 
, groups of students. The researcher treated 
'-■^ach the individual rroupc as k Judge of 
inath-^natlcs teaching perforriance in the ^ 
first JAN investigation. The second group- 



ing resulted 'in 10 groups each of w^iich . 
wa^ treated as a Judge in the secor^d JAi: 
analyses j The third grouping of students ♦ 
was made on the u-asis of mathematics: 'in- ' 
structors. Thi^ >grouping resulted in 26 
groups. vrhLch v^ere treated as indivicjual 
Judges in the thrid JAN analysis. 

JAN utilizes a multiple regression rro- 
del and a hierarchial grouping technique 
whici^ clusters reaters on th'e basis of the 
homogeneity of^their predication equatfonsr 
The procedure relates each- of k dimensions 
oV th^ stimuli (tn^f predictors) to the re- 
si^onse ^(criteria) made by k rater through 
a least:*-sauare,s 'regressiOR'.:-eaua>XQXi.»_ Once 
ratings for all stimuli have b^en obtained, 
the JAI3 technique uses an interative' me- 
thod of clustering criteria which retains 
optimum- predictive efficiency. 

As described Jby Houstort, "the JA!^ tech- 
nique starts with the assumption tnat eacn 
Judge ha^ ah individual policy. It gives 
an H (multiple R coefficient squared) for 
the initial stage consisting of all the 
Judges, each one treated as an individual 
system. Two policies are selected and 
combined on the basis of having the most 
homogeneous prediction equations, there- 
fore resulting i-n the least possible loss 
in predictive efficiency. This reduces 
the number of original policies by one and 
and gives a new R^ value for thi§ stage. 
The loss in predictive efficiency ^can be 
measured by finding the drop in R*- between 
the two stages. The grouping procedure 
continues reducing; the number of policies 
by one at each stage until finally all of ^ 
the Judges have been clustered' into a sin- 
gle group." (6) 

Following the> procedure used by ?Jous- . 
ton, the researcn'er examined the succes- 
sive drops in R-^ at each stage of ihe JAN 
process. The determination of whe:her one 
or more policies were present among the 
Judges "was made on^the basis of thS stage- 
by-stage drop in R"^. It was determined a 
priori by the researcher that a slippage 
greater than .05 between ^successive stages 
represented to great a loss in predictabi- 
lity. (According to Houston, a .05 drop 
in usually coincideSpWith a departure 
from linearity in the R drops.) By em- 
ploying this procedure it was possible to 
determine the Decetnber and May evaluations 
for each of the st^u^^h^grouplngs. 

Supplementing the JAN procedure the 
researcher used stepwise multiple regres- 
sion (BMD02R) in an effort to explain the 
policies identified by the JAN technique. 
The researcher determined the unique con^ 
tribution of proper subsets of the predic- 
tor variables, 1^8, to the prediction of 
the criterion (variable 9)« l^he contribu- 
tion of a set of variables to the predic- 
tion of the criterion can be measured in 
terms of thepdif ference between the full 
model (FM) R ^all predictor variables in- ^ 
eluded) and the restricted model*'(RM) R"^ 
(some variables omitted). This difference 
may be tested for statistical sirsnif icance 
^l^using an F-ratic or by establishirig an 
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VARIETIES OF INTERACTIVE COMPUTER-BASED TESTING 
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David B. Thomas 
University of iowa, Iowa City,^ Iowa 52242 



ABSTRACT: Con^juter- based mstruct^-on has spread from the obscure laboratories of the I960' s to often ^ 
extensive usa^ by colleges and universities, public schools, business and miustry, the military, and 
government. A recent accon^animent to the growing use of computers for instruction has been the develop- 
ment and use of computer- managed test ifem banks and computer generated tests. Further developments have 
m^^iuded procedures whereby students are te&ted while seated at computer terminals. This review will focus 
or. the varieties of interactive tests and on research which has suggested alternative methods^ for present-* 
ing and scoring tests presented by computer. Particular emphasis will be placed on a comparison of sequent- 
ial, branching, imear, and multi- staged item selection and presentation models for interactive cofeputer- 
bascd testirxj. 
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Compute r-ba^ed tests have generally been 
characterized as groups of test it<^s which are 
stored, constructed, or generated by means of a 
computer systq?^. Such is the focus given by the 
systems described ih a volume ecjited by Lippey 
(1974) m wl)ich test items reside m item banks 
stored within the^computer systems and m some 
cases on tab cards retained by individual instruc- 
tors. These models of computer assisted test 
construction provide for automated item selection 
(according to criteria such as content, difficulty, 
discrimination, and the like), item generation 
(using Item forms), and printing of tests and 
alternative for,ms. Statiistics relating to item 
and lest usage and item jlinalysis data are ai$o 
usually retained by the computer system for eval- 
uation of tbfei'items or for use by item selection 
algorithms. " 
«✓ 

An almost universal characteristic of the 
computerized test construction systems is the 
necessity for off-line administration of the tests. 
The applications for such tjest construction 
systems are widespread, and usage of the systems 
IS mercy's mg. Recent work has begun to extend 
the applicability of sQch automated systems to 
testing which takes pt^e at aji interactive term- 
inal so that the testing situation may be uniquely 
and dynamically structured for the individual 
student. 

An Interactive test may be defined for 
present purposes as a test ^administered at a 
computer terminal, such as a teletype or cathode , 
ray tabc display, m whiuh the student's response 
to each question is immediately evaluated by the 
corr^uter, a testing decision may be made, feed- 
back to the student may he provided, and another 
Item may be selected for presentation to the 
student. The interactive' proems is then repeated 
as often as necessary until the testing session 
IS concluded. A distinction needs to be made 
between the concept of an interactive test and 



some forms -of computer*assisted instruction (CAl). 
In certain varieties of CAJ test questions play a 
major Vole. Drill and practice programs co^isist 
almost entirely of questions; tutorial* programs 
intersperse questions with text; simulations and 
inquiry systems often use test-like questions as 
part of a dialogue. Groups of test questions such 
as these are not considered interactive teVfcs 
within, the present discussion even though these 
uses properly fall within the definitions provided. 
The interactive test as used here pro^vi^es guidanc^e 
useful between units of instruction whereas the 
CAI usage generally provides guidance within the 
unit. 

iJ the interactive testing situation the 
examinee is presented with a test item through a 
computer terminal. The response to the 'item is 
.evaluated as to its correctness and this inform- 
ation, together with other information which may 
be available (difficulty of the item, response v 
latency, prior test or item scores for the student, 
£or example) is employed in a decision model which 
may permit or select the next item to be presented 
or may even terminate the- testing session. 

The interactive testing models which have 
been developed may be classified according to four 
points of view representing the prdcedure employed 
as the student proceeds from item to item through 
the test. These procedures, or models, have been 
classified as (1) linear, (2) branched or tailored, 
(3) sequential, or (4) multi^-stagod. 

Linear interactive tests - The linear inter- 
active test is one which is translated directly 
from a papcr-and-pencil format to computer presen- 
tation, tn such a test an item is presented to, the 
student! and a response is elicited. The correctness 
of ^the response is determined either through refer- 
ence to a key or by com^Sarison of the students' 
response to a generated answer if the item is 
dynamically composed. A kdy chayraCteristic of the 
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linear test is the general absence eft a detfision 
function which would alter the length of the test 
or route the examinee from item to item. Items 
are ^i^esentfed m a predetermined order without 
exception. Linear interactive tests ape common — 
m computer-managedf instruction ICHi) courses 
such as those at FAorida State University and the 
University of Iowa. *tcf. Hagerty 1970). The 
student enrolled m CMI coyrsps at the Univer- 
sity of. Iowa signs on to a^' computer terminal, 
IS presented w^th a short linear test, th# test 
IS immediately s'cored an5 the student receives 
feedback as to whether tl^ instructional unit has 
been passed. In the Florida\ State University 
system, the same model is followed, except that 
the short licHear tests are cijmposed of items 
randomly selected from an itelh pool. A mbro » 
complex mode of interactive testing imd one 
-j^roVidmg a higher degree of individualization 
IS the branched testing mode. 1, * ' 

Br anched testing models -TOSt iVems presen- 
ted undfr a branchec' fornpt res(\mble the form 06 
irvf:era^,Mon seen in a programme^i instruction text. 
The student responding to a branched t^ist is 
presented with an item, an answer xs elicited, 
answer is evaluated, and the nOxt it*m is 
selected for presentation on the ba'sis 'of^the 
answer evaluation. Typically, they>ext item 
selected is determined cin the basi^. of ittn 
aifficulty. The difficulty level o^ the pr*iyious 
itera and the .examinee's answtjr are determinants of 
the 'cJitf iculty of the succeeding item. It is 
desirable that items employed m aptitude and 
ability tests haye^ di f f icul ty levels of approx- 
imately . 50 (about 1/^ the examinees, answer the 
Item correctly) for ^he population of students for 
whpm the test is designed. For the individual , 
the objective is to design a test with difficulty 
level nearly the same as the examinee's ability, 

^ and to do so m a way that the test has difficulty 
level of .50 for- that individual. The branched 
te^t IS design^ to do this by presenting a more 
difficult Item after an it(|ffi answered correctly 
and presenting a less difficult 'item following 

I Ain Item answered incorrectly. The litems m the 
Item pool making up the branched test are cali- 
brated to determine their difficulty levels prior 
to thoir being administered to examinees for whom 
• branching is to occur, ftn bxammee typically 
begins the test by a;^swering aji item of mod'erate 

' difficulty. 

Withm this framework, Bayroff and Seeley 
(1967) constructed branching testi of eight and ' 
nine items for use with Army enlisted men. The 
initial Item presented had a difficulty level of 
.60 and each subsequent item differed from the 
iteb preceedmg it by .05 on the difficulty scale. 
These investigators found correlations between 
scores on the branched tests and 40-50 item linear 
tests of .79 arid .83. Reliabilities were found 
to exceed those predicted for the linear tests 
of <;imilar length. Hansen (1969) constructed 
similar tests for use with students m a coUege 
physi'cs course-. High reliability was reported 
as well as a reduction in testing time. 

* 

If prior information ro^ardiniN^an examinee is 
twa liable, sue h^ as a teacher rating, relia'ble self 
tr'cports, or prior test scores, then the d>friculty 
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level of the first item to be administered may be 
selected m a manner consistent with this prior 
information. Weiss •(1973) describes a branched 
testing procedure (stratif led-adaptive or stra- 
daptive) m which items are grouped ^nto nine 
strata representing approximately equal d,ifficulty 
levels. /Items of difficulty level approximately 
.05 form the first stratfura, items with difficulties 
of 'about .15 form, the second, and sa on. The 
examinee is first administered an item from the 
stratum closest to* the ability level suggested by 
the priqr information available. Correctly answer- 
ed Items are fallowed by administration of the next^ 
available unanswered item selected froirt the stratum 
of higher "difficulty. Items answered incorrectly 
are followed by t^e next available item m the 
lower difficulty stratum. Testing continues until 
both basal-^'d^ ceiliftg strata are identified. Over 
k dozen scoring formulae are' being investigated to 
derive the actual ability level obtained.^ Weiss 
has shown* that for the stradaptive test the dii 
culty l«evel for the individual is very nearly the 
optimal 1-evel of p = . 50. t ^ \ 

Other procedui^es for branched^testing have 
been proposed,^ »some involving paper-and-peqcil \ 
vorsions utilizing programmed booklets an'd special 
answer -sheets. Alternative item selection strat- 
egies have also been^mvestigated, notably Bayesian 
methods* 

1 Sequential testing models - The branched test- 
ing models described previously are procedures for 
routing the examinee to items m a test which are 
appropriate to his or her ability .level. With the 

, exception of the stradaptive tefet, all of the tests 
described are of a fixed length. Even th6ugh the 
Item makeup and branching rules differ, a specific 
test will typically be designed to icontam a fixed 
nlimber of items for each examinee. \ Only the stra- 
dtptive test* empl<^ys a termination x^ule which 
results m tests of potentially different length* 
The sequential testing models to be described 
employ either a 'fixed item sequence or a rule for 
q^eneratmg bomogeneous items. A scoring and term- 
ination r^ule delimits the number of items present- 
ed during the interactive test. 

The sequential testing procedure is inplement- 
ed as follows: th^ student is administered a test 
Item; the response^ t9 the item is evaluated accord- 
ing to the specific requirements of the model; the 
examinee is tentatively assigned to one of a 
number of classes representativ|J of the ability 
estimate obtained at that pomtl ^The risk of *^ , 
incorrectly assigning the examinee is evaluated 
. m reference to the amount of information avail- 
able at that point m the test, and to parameters 
supplied by tho test administrator. If the risk 
of assignment to an incorrect class is acceptably 
low, the test Will be terminated, but if the risk 
exceeds an acceptable level another item is admin- 
istered and, the 'decision process is continued. 
Three ma:)or differences betweeh the sequential and 
branched tests should be apparent 
number of items m the sequenti 
compared to the generally fixed 
the branched test. Secondly, the items themselves 
are fixed in tho sequential teajt. Every examinee 
who completed ton items, isay 
presented the same items m tho same order (unless 
f 
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number o-f items in 
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homogeneous items have been generated). Thirdly, 
whereas the branched tests are designed to direct- 
ly estimate the examinee's ability Xevel, on a 
continuous scale, the sequential tests arc design- 
ed to classify the <?xciininee into a group which is 
representative of the ability level. Alternat- ^ 
ively, classification* may be made into groups of 
other kinds. Examples of such groups are pass- ^ 
fail, high-middle-low aptitude, accept- reject, 
program 1 - program 2- program 3, and so on. 
For many applications, such class designations 
may be sufficient and perhaps preferable to 
traditional ability scores. 

^ A sequential testing model has been developed 
liy Thomas (1975) which employs repeated applica- 
tions of discriminant functions to classify exam- 
inees into one of four ability groupings. As 
itcmb are presented to the student, a previously 
derived discriminant function is entered using the 
,-wm{'ietu 'response protocol for the present and all 
previous Items. The discriminant function indi- 
..atts. the class to which the examinee most likely 

-^•houid be as»i>igned, given the response protocx?l at 
that point in time. When a small number of items 
have been presented, the probability associated 
with the "most likely class" is typically small 
enough so that one would not be highly confident 
uf a decision to assign tine student to the identi- 
fied class. Thus* the probability of membership , 
in each t^lass is employed as a termination rule. 
After each item has been evaluated, the highest 
probability li.e., that associated tWith thp "most 
likely class") is compared with a minimum accept- 
able probability. If the minimum is exceeded, 
then testing stops; otherwise testing continues 

, until some previously specified maximum number ' 
of Items have been administered. Thomas found 
that only 1/2 to 1/3 the normal number of items 
were required 'tor cla&sif icatipns and that 70% 
of the examinees were correctly classified. 
"Major" errors ^ 'classification occurred for orrly 
5\ of the examinees. i 

Ferguson (1971) employed a hierarchiel model 
of criter-jLon-referenced roeasuri2ment in cDi,3\iction , 
wj:th a sequential probability ratio test Wald, 
^947). Homogeneous items were individyaLly ' 
qoner^ated through the use of item forms. Exam- 
inees were J evaluated relative to course lohject- 
ives by administer ing an item and appXyiii^ the 
sequentiall probability ratio test to determine 
whether thp student had achieved mastery, 
failed to (achieve mastei^y^ or whether judgment 
should be suspended. A nor^mastery decision 
terminated testing.' In the case of suspended 
judgment, another item was generated, the item 
was admin4St;ered' to the examinee, and the seq- 
uential probability ratio test was repeated. 
A mastery decision led to testing of the next 
objective in the hierarchy. As a result of 
^having to test fewer Objectives in the hierarchy, 
the author fbund high reliability and a total 
testing time reduction. 

Lmn, Rc^ck, and Cleary (1972) employed a' 
simJl^r njqdel qf sequential , testing with three * 
scales of the CollegQ Board's College Lev'bl 
Examination Program. The i^ems were administered 
in the same order as in the original scale. 
Those jiuthors found that one-half the traditional 



number of items were required for the sequential 
testing model to ^entify students who. should be 
pla^ced into one of two categories. 

The sequential testing models have not enjoyed 
the attention nor haVe they been the subject of as 
extensive research interest as the branching tests.^ 
This is primarily becavfse they are inherently con- 
cerned v^ith selectijon rather than measur<^ment, per 
se. The sequehtial procedures as a group require 
the development of fewer items than the branching 
procedures and may ultimately emerge for this 
reason. P,erhaps the selection function necessary 
for the multi-staged tests, described next, will 
prbvide the stirijulus for more interest in this 
prpcedure. ' 

Muj.ti:-s'taged models - The final variety of 
interactive tests to be presented are the multi- 
staged tests. This testing strategy requires two 
or more sub-strategies on sub-tests. The first 
stage consists of a routing test which is admin- 
istered to the examinee to ascertain a general 
locrel .of ability. This level may be characterized 
as broadly as high-middle-low, or may be somewhat 
more precise in nature. The routing test is 
typically followed by a "measurement" test which 
IS peaked^ at the ability , level indicated by the 
routing test which preceeded it. Characteristic 
of the multi-staged tests are those which have 
been investigated by Cleary, Linn and Rock (1968) 
using -3imulata,on techniques and existing item data. 
Four procedures were studied which divided examin- 
ees into four ability groups. Each of the four 
ability groups was then administered the measure- 
ment test of the appropriate ability level. The 
four routing procedures were: 

1. Two-stage routing test^ - Tpn items were 
selected, which had difficulty values of 
approximately p = ^50. The examinees wtfre 
divided into two groups on t)\e basis of 
scores on this first ten-item routing test. 
Each group was then administered a 'second 
routing test of ten items selected in the 

, same .manner within each ability rc^gp. Four 
ability groups.were thus. formed which could 
be administered the appropriate measurement 
t/ests. I 

2. Broad range routing test - Twenty it;ems were 
selected which represented a broad range ^of 
difficuljty. Examinees were administered 
these twenty items and divided into four 

• groups on the basis of scotes. 1 

3. Group discrimination routing test - Twenty 
items were selected which showed the largest 
range of within-group difficulty indicjss 
(i.e., difficulty values computed for [each of 
four ability groups). The score <Qn these 
twenty items was use\j to form four abijiity 

' groups. 

4. Sequential routing test - Twenty-three items 
- were selected for use in a sequential test 

as described in the pre\iious section. Four ' ' 
ability classes resulted. ' • 

These investigators, in a series of titudies, fiave 
showed that a procedure of sequential routing to 
the measurement test showed highest correlations 
with the total test Score, but that the group- 
discrimination routing test showed higher correr 



lations wxth external priteria such as achi^evement 

• scores m History and English. - 

Application or Research? ■* T^e varioties of, 
interactive computer-based* tebtinq which have been 
reviewed *a re largely confined to the restjarch lab- 
oratory.. Researchers are not j recently j^re^^ared 
to recommend specific soodels €<n wide-spread 
adoption - indeed could- not do su without signif- 
icantly ex^nding research with studer^ts in a non- 
laboratory setting. The interactive oomputer- 
based testing which does occur withir> the context 
of ongoing instruction and which is relatively 
free from user concern for examinee welfare 
relatJb to one iiKxiel only: the linear test. Yet 
even the linear test is not free of affects 
related to tho computer tormirjal situation. Some 
research, for example, seems t*) indiLdte /^hat 
black stiidents perform better cn a terminal- 
; criented test than on a tVaditic^nal ly administered 
test. F.iridirigs of this nature lead to specula- 
tions as to the equivalpnce of alternate forms of 
the test 'and to the errors of ntjOi'^^i^ement - 
whether stimuidted by .om^uter or by paper- and-^ 

♦ pencil testing. 

The ust; of computers for intcra>^tions of aoy 
kind with students .lead tc considerations, of costs 
vs. benefits. Consideration 6f the relative worth 

of teacher and student time also needs to be made. 

In thos^ situations whert ,i>tudent time is impor - 
tant and costly, as *in the military, for example, 
computerized tests may be administered on demand 
within an individualized ins^tru^tion framework. 
Even if administrative costs are not lessened, • 
cost-ef f<ictiveness may be improved. 1 

In situations where many different tests are 
given or where fast scoring ic> desirable, 
computer-based tests ma^ be an attractive alter- 
nativ<*. Where testing occurs at different times 
for different students and item Security may be 
a factor, the computer izod test may be valuable; 
an Item on a CRT display leavt^s no "hard copy 
for posjSible compromise. 

iTi summary, let it kje said that computer- 
based interactive ttbts aru bvinq actively 
explored by mea<^uremerit c;pt^c lai'i sts, educational 
researchers, and instructional technologists. 
Muc^i testing has been taking i lace cit completer 
termmaK^, primarily' of a linear sort, and the 
trend to \ncroas*^ computerized t< i^tmg appli,'a-' 
• 'tions will continue. Th«; tecjhniviue*; described 
here, 'which indi vidua Lizo tht testing inter- 
action, will certainly be do^'eloped to the stage 
that will permit th^ir. oduca .loHpl use within a 
very few years. . I . 
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' THE CONSOLIDATION OF SOCRATES*: 

r 

GROWING PAINS IN A LARGE COMPUTERIZED TEST GENERATION NETWORK 

c 

Lisa Gray-Shellberg, California State College, Dominguez Hills 
W. Van Willie, Caliiiornia State Univefeirty, Fullerton 
Oliver Seely, Jr., California State College, Dominguez Hille 

State-Wide Project Directors ^ 

SOCRATES Computer Aeeieted Test Construction 



Substantial increases in the number of users, number of item banks, and item bank sizes have occurred in 
the past year. These changes, coupled with feedback from teachers and support personnel, have provided ^ 
valuable input for consolidating 30CRATES from a pilot or demonstration phHse to a routine operational 
phase. The experience gained in this task should provide invaluable input to, educators interested in 
developing and implementing cpmputer based test generation, scoring, and analysis (CTG) ]{)ackage6 for class- 
room support. Five areas of experience are discussed as especially important to any such project: design 
of item banks for maximum selectivity and item retrieval, design of items and processing them into the banks, 
user attitudes, habits, and training, hardware and software constraints-, effective use of CTG as a class- 
room resource. 



Introduction 



Notice!!! Test preparation is no longer a labor- 
ious task. The timesharing Exam Program enables 
an instructor to prepare a test by entering the 
following simple commands iilto an office terminal. 

Exam test request (identifying the test genera- 
tion Program) 
General chemistry (identifj^ing the question bank 

to be used) \ 
Stoichiomj^try, volume calculations - 10 items 
(identifying category and Quantity of items to 
T&e selected) . \ * ^ ' 

Chemical equations, predicting products - 10 items 
Nomenclature, binary Compounds - 10 items 
Redox, determination of oxidation nximhers - 5 
items 

Keyword: test 23» chapter 3» ^ ({questions to be 
limited to those from a particular tjext and chap- 
ter) f 

Miied difficulty levels, k versions 

The screen 'displays the requested items se- 
quential^ly, allowing the teacher to accept or 
reject them. Five different output options are 
provided: (l) the test may be placed on floppy 
disc at the test center for students taking the 
course on a self -paced basis; (2) the test may 
be printed on the electrostatic copier in the 
department office (1-10 copies)^ (3) the teat may 
be printed on spirit duplicating masters at ,thc 
teacher's office terminal with a 50% linear re- 
duction (11-2<X) copies); ik) the test may be 
co](ied automatically onto microfiche cards with 
trl-color diagrams to accompany the text of the 
questions (11-200 copies); (5) the test may be 
transmitted to a <que*que at the duplicating center 
to be processed on its computer controlled mul- 
tilith machine (100-5000 copies). 

\ 

Sound like science^ fiction? It is if the] 
teacher is employed by any public or private 



stitution of learning in the U. S. Yet the tech- 
nology required to retrieve tests and deliver them 
by any of these five options has been available 
f or ^over a decade. 

The development of SOCRATES J)egan in 1972. 
Since that time many people have contributed to 
its growth. Our task this year has bean to con- 
solidate /the progress to Mate. We have observed 
certain problems associated vi(.th the application 
of computer technology to the educational process. 
We present some of our experiences here for others 
who plan to develop a similar test generation and 
scoring service. 

The teaching/learning process contains two- 
major components: (1) the delivery of subject 
matter to the atudent and^ (2) the evaluation of 
student progress. Although the number of ways 
in which these two components dan interact is large, 
the relationship is often in one direction only; 
the nature of the second is detcfrmined by the 
nature of the first, but no^Svice-versa. Follow- 
ing, a period of learning, ^theXstudent 'is examined 
for acquired knowledge. The examination ^rves 
as^^an evaluative tool on^y anduiot as a guide for 
the delivery of roorf course mattrial. As long as 
the level of delivery is aimed st the average 
student the exam§ are normed\pn the class 
average^! this jnethod is considered to be 'a success. 
That some students exqell and others fall further 
and further behind ift considered simply to be a 
peculiarity of "the system". Over the years, this 
commonly observed shprtcoming gained gre«iter promi* 
nence and educators began to suggest methods that 
wefe more sensitive to students* needs. 

Today we see many outgrowths of this early 
desire to improve the system: programmed in- 
struction, computer assistecKJ-nstruction*, self- 
paced learning, personalized iijstruction to name 
just a few. All of these method/3 tend to focus on 
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the students' needs an determining the lev«l and 
the speed of course work. Unfortunately, they 
have the fault of requiring the teacher to 
spend excessive amounts of time preparing and/ 
or evaluating course work for students at various 
levels of progress. It is the evaluation of stu- 

progress which proves to be j>articularly 
burdensome: most of the personalized methods 
use n modular approach, with each module being 
concluded by a test over its contents. Some 
teachers administer a pre-test and a post-test 
as well OS repeated post-tests until p r^rtain 
mark is attained. It is here that the computer 
offers a solution to a very real educational 
problem. It can generate and print tests quiclcly, 
score thf^ results, maintain student records, 
and compile item statistics for future test devel- 
opment. 

Cf course, educational administrators 
ask for mor.- than that a new method, medium, or gad- 
get be qble to solve or alleviate a recognized 
educational problem. First, it must be just as 
acrfBsibli as the method earlier used for the 
same purpose. Second, it must be competitive in 
price. Th^'se two additional criteria pose larger 
problems. 

A companion of manual preparation and scor- 
ing of eounl numbers of exams, vs. computer gener- 
ation andyscoring will doubtless show the auto- 
mat i<;^- method to be cheaper. To compare the genera- 
tipti and sroring of large numbers of tests for in- 
^dlvidual students, even by the computer, with the 
manual* assembly of perhaps, a one-hour midterm 
ej^amination and a three- hour final examination 
each term, the cost comparison becomes somewhat 
less favorable. There is another problem. Even 
if one were to compare the cost of generating and 
scoring equal numbers of examinations by bpth 
methods, the cost of the computer generation is 
added to*»existing educational costs rather than 
replacing them. 

. The teacher is still paid the salary of a full- 
tame faculty member. The secretarial staff is not 
reduced. That the teachers and secretaries can ^ 
now redirect their efforts toward more useful en- 
deavors thereby improving the educational process 
cannot be denied; it is not, however, an argtunent 
for reducing costs. 

Then, there is dhe matter of accessibility. 
In the early days of this century when the auto- 
mobile first made its appearance, people worried 
about the problem of refueling on long journeys. 
The automobile enthusiasts of the time v^cnt out oh 
a limb in predicting that refueling would be no 
problem at all as someday there would be one gaso- 
line station in every large city. 

Most educational institutions, in contradic- 
tion to the state of the art, find themselves sHU 
very much at the beginning of a new age. It is " 
one thing to paint rosy pictures of automatic test 
retrieval as we have at t^e beginning of thid 
article but quite another to attempt to convince 
financial officers of the educational efficacy 6t 
computer, assisted test construction 'when the system^ ^ 
is plagued by any of the following problems: 

TJheNterrainal which prints the tcVts is more than' . 



five hundred railed away (for some of our campuses). 

The terminal which prints the tests is half-way 
across campus (for the rest of our campuses). 

Your local Campus* ADP director .will support the 
printing of tejsts on the local high-speed printer, 
but will'S^not supply them on continuous form re- 
production masters. 

The reproduction of tests printed on white paper 
requires an advance notice of two weeks at the 
duplicating center. ( 

The local campus ADP director jrfill not support the 
SOCHATES system because he is already understaf f<?d. 

•The test request (telephone-oral) gets garbled in 
transmission. 

The test reouest (ptmch card via cable) is miE 
punched. 

The computer breaks down-, f ' «a* 

The ADP director is supportive of the system, but 
the print chain on his terminal is the wrong one. 

The teacher discovers during the duplicating pro- 
cess that the ditto masters were mounted backwards 
on the high-speed printer. 

The test arrives on standard white forms, but the 
print IS too light to produce decent optical mas- 
ters. ' 

The test arrives by first class mail ten days after 
,i.t was postmarked. 

The Organizational Structure of SOCRATES 

^The Division of Informat4,on Systems of the 
California State University and Colleges i& cen- 
tralized at the Chancellor's Offite in Los Angeles. 
A- dualCDC3300 computer services the needs of all 
nineteen campuses,* though eadi campus has a smaller 
computer which can handle small jobs with quick 
turn-around. The organizational structure of pro- 
ject SOCRATES is shown in Figure 1. The role of * 
the major components in the diagram are described 
*b*low. 

User-Contributors 

There are slightly over three hundred users 
Registered to request tests througfi SOCRATES. 
These users are distributed* over the nineteen cam- 
puses of the California State University and Col- 
leges. The potential user group within the system 
IS roughly sixteen thousand. If the SOCRATES sys- 
tem remains centralized an^ its use is opened to 
•faculty of the community colleges as well, then 
tlT^s potential user group would expand to over 
J:hirty thousand individual teachers. The daily 
request rate varies from one to three percent of 
Nthe total registered users. In the month of Nov- 
embers, 197$, there were 151 test requests processed 
Users request testa from banks covering sixteen 
subject matter areas in one of two st^fsi directly 
^ by telephone to a clerical assistant hired to take ' 
» test^^j:^equests the*<;haneellor^6 Office or by" 
remote- jbb-ehtcy At one'd local ciampua cb;nputer 



center* The component labeled *'cofflDnmicati6n6 
interface** accounts for either of these functions* 
From the very l^eginning, our users were encouraged 
to Qubmit thein own items to existing banks and to 
develop new banxs in other disciplines* As con- 
tributors, they are exposed to the finer details 
of cross-classif icationt categories, keywords, 
difficulty levels, behavior levels and special 
characteristics* Six of our existing banks and 
five new ones have been built by faculty who wel- 
comed this opportunjity and immediately saw advan- 
tages of using computer assisted test Construc- 
tion in^their disciplines* i 

Sank Coordinators 



Excluding the U* S* 'history bank which * 
was developed by a large number of teachers in the 
Los Angeles Unified School District, each SCXJRATES 
bank was developed by one or two faculty members 
in the CSUC system* Developmenjf included writing 
appropriate category indices, gathering items for 
inclusion, clacsi fying the items according to cate- 
gory* difficulty and behavior level and -certain 
cross-references* In the early stages of develop- 
ment, SOCRATES bank coordinators were from our ^ 
Southern California campuses* Only recently, as 
the system has begun to 'be better publicised, are J 
coordinators from the northern campuses beginning' 
to .volunteer their efforts* After a bank h*a6*|^assed 
through its initial development, users begin to ask 
, if thoy may submit their own items to the bank* 
Dedication to this task varies greatly, but' the 
overall effect is to change the role of the bank 
coordinator from innovative developer to that of 
manager* As new groups of it 6ms are received, they 
must be logged in and checked against earlier con- 
tributions to guard against duplication* Keypunch 
operators need to be kept informed on changes in 
notation* /conventions for each bank* As ^he volume 
Vf su^itted items increases, coordinators find 
ir'fmpossible to undertake the task of coding them; 
those faculty who have submitted items are gener- 
ally assigned this task* It is' the coordinator's 
job to keep these individuals informed "of changes 
in the category index and conventions on keyword and 
special characteristics* Finally, it is the coor- 
dinator's job to set standards of reliability based 
on the Item statistics and periodically to edit 
the bank on the basis of these standards* 

Campus Coordinator s 

Each campus has a testing of ficer whose tradi- 
tional duties have included the administration of 
personal aptitude inventory tests, the coordination 
' of periodic college board exams hel(} on the campus, 
advising the local academic senate on the establish- 
ment of standards for teaching effectiveness surveys 
and' assistance to faculty with research requiring 
expertise in the area of tests and meas\irement* 
The development of project SCXJRATES, the avail- 
ability or automatic scoring packages locally and 
thp renewed awareness amon^ faculty toward relia- 
bility of tests"}ias given these testing officers 
the added respoi^sibiJ^ity of becoming the local 
SOCRATES expert; The response from the testing 
officers ranges from .enthusiastic support to pre- 
meditated Ignorance* The former generally see the 
role bf their office* to be that of service to 
faculty, ^fhatever the need may be* The latter are 



often employed by the counseling centers on their 
campuses and they traditionally have not been involved 
m classrooi^ testing* Nevertheless, as SOCRATES 
grows, we shall look more and more to. these indivi- 
duals to take on the task of maintaining a local 
supply of documentation and assisting faculty to 
order tests* 

Project Directors 

As the SOCRATES system grows, the role of the. 
project directors changes from that of iijspired ^ 
(possessed?) innovator to that of evangelist* In 
the beginning, it is necessary to secure bome pro- 
gramming support to write the prototype retrieval 
system and a few dedicated (hoodwinked?) ^aciMty to 
develop the first item banks* After the f)|.rst if ew ^ 
banks are operating, an^ with the assistancye oii the 
workshop coordinator, the directors travel from! campus 
to campus, spreading the gospel and distributing \ 
dumps of item banks like Gideon Bibles* \ « 

Systems Analyfet - Progreumners 1 

Project SOCRATES employs one part-time system^] 
analyst and a full-time programmer* The systems 
analyst 'and the project directors together evaluate 
ddsires and suggestions made by users, bank t:oor- 
^inators and,ADP directors on the various campuses* \f 
After much^discussion and establishment of pnor- ^ \ 
ities each year, the programmer tackles the job»(^f 
implementatioz^* The programmer also serves as tr.buble- 
shooter when requests for tests fail for reasons pe-^ ' 
yond the knowledge of the -clerical assistant who \. 
takes the request, or the user who submits the re-t 
quest by remote- job-entry* \ • 

Local APP Directors \ 

The SOCRATES system represents a very minor part of 
the job responsibility of ^the local ADP directors,, 
whose, attention is focused on supporting the regis- 
tration, accounting, grade reporting, and a broad ^ 
spectrum of instructional computing services* .1 . 

^Nevertheless, most ADP, directors see SOCRATES a^ a 
way of giving instructional computing a larger voice 
in the determination of campus policies affecting 
computer services* The ADP directors, through 
their support personnel assist users^ in generating 
examinations by remote- job-entry and having them 
printed on schedule at the local campus high-speed 
printer* The bad. news is that there is a wide- 

• spread reluctance to have SOCRATES tests printed 
locally* Faculty are reluctant because the local 
character set is slightly different from that used 
at the Chancellor's Office; ADP directors are re- 
luctant because the line speed is only baud — 
any significant* use of SOCRATES by faculty would 
swamp local campus systems* Practically all ADP 
directors refuse to support the l9CaI use of ditto 
masters, partly^ because of the expense .involved and 
partly because of the static the^ got from their 
operatbrs* 

Summary ' 

The reader might wonder what has happened to , 
the rest of the connecting lines of command in 
Figure 1* The answer is that there are none» 
Each circle in the diagram represents a person who ^ 
works for and is paid by a different administrative 
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unit# The bank coordinators are largely faculty 
members on eeparte campuses, unpaid for their de- 
dication to SOCRATES. The same is true for the user- 
contributors.f The keypunch support for this year 
has been donated by the ADP director on one campus. 
The project dlirectora and workshop coordinators 
are employed/ by the Office of New Programs and 
Evaluation. / The computer operators Nare employed 
by the Division of Information Systems* The campus 
test officers are usually professors of tests and 
measurement on the various can^uses. The systems 
analyst nni programmer wcrk for the InstructiSfial 
support Gidoup of the Division of Information Systems. 

The fedmiLstrative structure of SOCRATES is 
a mana^en's nightmare and an academician's dream. 
With tho exception of the Tceypunch operator and the 
computer operators, who answer directly to their» 
supervipors, everyone else is free to do as he or 
che pl^nnes within some very loose philosophical , 
boundarien ^nd the programmed constraints of the 
TiyBtom. Th* campus testing officers are free to 
onf"i ir»re or not tOr encourage faculty on their 
rnmpuR'ft to usfc the system. The bank coordinators 
aro f roe to use* many forms of cross-references in 
developing toeir banks or'none at all; they are 
free to work closely with users who i^iah to con- 
tribute their Own items or to let those users work 
directly through the project directors in getting 
their items coded and loaded. Users .are encouraged 
to use existing banks and to develop ^ew ones in 
different disciplines and specialties. Although 
they are encouraged' to use the SOCRATES scoring 
package in order to add to the" item statistics 
file, they are free to use scoring packages avail- 
able at their own campuses or to use more accessible 
self-contained scoring machines. The workshop 
c cord inatpr -Maintains an active schedule of jmeetings 
with interested facujlty on the various canpuseef, - 
assisting in the request of initial tests, The 
project directors maintain an-office which" serves 
an n clearinghouse of ideas and documentation. 
Suggestions: */hi<:h are deemed suitable for serious 
COTsideration are discussed by pl\ concerned at a , 
monthly .coordinator's meeting. The offirce maintains - 
current cbpiee of all system documentatiipti for, 
users, coorxiina^ors and interestefd* correspondents 
alike. At this early stage, the directors' job is 
to extend an open invitation to faculty both to * ^ 
participate and Contribute to the banks.' 

There are at least two. advantaged to such a 
loose structure^ First, the system will grow 
through faculty initiative rather th^n admini^e- 
trative dir,ec^ve. The charge that 'SOCRATES is* 
bein^ used, as aVnieans of controlling .^11 classroom 
curricula centrally will be groundless^. Secondly, 

expansiQn.of the systeA to includ^^ the* community. 

colleges of California vljien the .piiblic agency data , 
processing network is^ implemented will require little 
'change in its- administration; there Will be some 
now banks, some extra money for hiring addftlonal 
support -personnel and a greatly expanded group of ^ 
user-iontributors. There ip etill widespread un- ^ 
certainty an^ outright f^ar and ignorance among 
faculty and administrators about applications of* 
♦ computer technology to the educational process. ^ 
' The development ot systems such as SOCRATES is 
better served 'by a loose structure, because its 
slow, natural growth ^^rea^gs no one* It has 
the added advantage of bt^w^ \is time to improve 
the aystem. 



On the other hand, it is imperative that we 
recognize that the key to success is to continue 
to strive, through effective management, to achi^eve 
"the greatest good for the greatest number". The ^ 
question IP not whet>ier to give permanent support 
to computer nsjsisted test construction, thereby 
assuring its continued development. The observaCToh 
that the rompUter has become an indispensablii tbol 
m supporting routine operations of educational 
mctitutionr.. including the maintenance of records 
on registration, grades,' library holdings, and pay- 
roll, forrpfi one to realize that the real ouestion 
IS whether }o allow computer assisted test construe- , 
tion to develop on nineteen campuses in nineteen 
different ways with hundreds of programmers duplica- 
ting eacliTother's efforts and thousands of professors 
subitiitting tens of thousands of test items, many of 
. which are the same items that are being submitted 
by their colleagueh, to4jundredr. of ksypunch operaittrs 
all punching them in different formats. Can we 
afford to let profer»sors keypunch their own items 
to b( used in their own private banka with primitive 
retrieval systems written by them in the evenings 
and on weekend.*; with no coordination with a single 
other professor in the whole CSUC system? , ^ 

That the final success of, computer assisted test 
construction ib inevi^ble, ^ there^ can be no ouestion, 
judging from the increajsing demand for more rapid ' 
and efficient info.rmation retrie^l from mapy sec- 
tors of society.' Its usefulness will continue to 
be hampered though until th^ computer becpmes_as 
relia\ile als-'the department secretary or the office * 
^-typewriter,,' until teachers place the same trust in 
^the comp,utei* to. produce tests as -they do in a text- 
^book t a produce foii test questions and until 

'users begin to demand the same- quality of info^ 
raation retrieval service as the^ can now get from 
theXr^ campus libt^riahst . , ' * 
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N=300 now 

N=^,000 faculty ^ 

1,000,000 studenta » 
^ (potential^ ' \ 

o o ^ ' . 
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Bank \ ^ ( potential > 
Cobrd* 
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ABSTRACT:' The objective of computer-assisted tes ti n/"i n 'thermodynami cs is to present ^ 
questions as- efficiently and as accuratel^y as poss^b/le, with the minimum feedback to the 
s-tudents at time of ques|;ion presentation and the fna\$jmum evaluation of student's knowl- 
•edge in thermodynamics. 'Questions from a computer teVisis Selec-ted from the computer- 
assisted- 1 nstructi on in thermodynamics de^veloped by the artjthor and o^ther professors at 
U. S. 'Naval Academy from 1 9fr7 to 1971. The questions can be selected by students on the 
basis of be^tavioral objective^,* Feedback is given at the end' of the test as to what the 
student missed and what work needs to be prescribed for him. The more feedback loops 
used in the* testing t\r^ram, thd closer the testing approaches to computer-assisted 

instruction. . 
a ' 



1 . I ntroducti on • ' i' 

From 1957 to 1971 , the Division tif \ 
Engineering and Weapons, together with 
the Academic Computing Center* of the U. 5. 
Naval Academy engaged in a joint Computer 
assisted Instruction projectiU ta pro- 
duce teaching materi a 1 s, i n-^ thermodynam i cs 
for adaptation on the IBM 1500 instruc- 
t^xjnal system. »Four experimental sec- 
tions consisting of soxty-two midshipmeh- 
tauqht by the author an<i Lieutenant Corr- ^ 
nander ,M. Tuft used segm.ents of the con- 
pleted CAI material in the thermodynamics 
course during the 1968-1969 and 1969-1970 
academic vears. The project was set up 
i n three phases : 

(a) Development Phase - To be'used 
to v/ri'te all the CAI materials and to 
p^roqram them into the^ comput>er . The 
prejpara ti or\ of each STnit of the course 
naterial in i ns^tructi onal form is de- 
'scribed by Wu a'*nd Leel'M. 

(b) Valid^ion Phase - To be used 
to co'nduct classes, to determine 
whether the objectives are meti, and^o 
rfiake fevis1t)ns where needed( <; 
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(c) EvaluatioJ) Phase - to be used 
to conduct classes, to determine the 
CAI pres'enta ti on as compared to the 
oon-CAI presentation, and to malke re- 
visions and recommendations r^gardinq 
implementations- A general surVeyv'*) 
and a statistical survey(^) on the CAI 
thermodynamics hav^e been reported. 



le CAI thermodynamics course has 
specified to be "-totat immersion," i.e., 
all the raaterials of «the course are to be 
taught bf'CAI. HoweveV, the results of 
the studJents' performance and responses 
Jn\the C[AI clashes' and the experience 
*gai\?ed ty the Author(^) and other pro- 
fessors/who taught CAI or non-CAI thertno- 
dynamiqs from 1967 to 1 97J ' po i nted •to the 
fact tHat due to limitation of the CAI ^ 
materii*! authors' time, the CAI instruc- 



t i ona 1 
as the 
the CO 



material could not be as thorough 
professors desired, thus causinq 
icept of "total Immersion" of the , 



CAI thermodynamics course to be dropped. 

To make use of the existing CAI ther- 
modyrtamics material, students who enroll 
in tfje thermodyjiami cs course will be en- 
couraged to take testing pf those seq- . 
ments i/i which they may experience diCfi-1 
Cul ti es . . ^ 

2. Equipment 

*■ ^ ^ * 

The computer-assisted teachlhq unit 
U 'the I-aM 1 500 system which is comprised 
of an 1800 process .control computer, a 
1 5021 multiplexer, five 2310 disk drives, 
two 2400 tape drives* a 1442 card reader/ 
punch, and an 1132 pri-nter. 

Eath student station consists of: 

, \ 

ial) A 1 iijht pen and a typewriter key- 
board,*' whici allows^ a midshi pman "to con- 
trolt the machine's 'presentation of feest- 
inq tiuestions and al<;,o to submit his an- 
swers to the testing questions'- 



1. 
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, (b) An ima^e projector and a 1515 
cathode ray tube which enables the sys- 
tem to communicate with t he ni d s h i pmen . 
The testing questions are presented on 
the cathode ray tube and figures or 
graphs are displayed by me*ans of the 
1512 ima^e projector. 

3. Testing Logic 

The computer- a s s ii ted testing mate- 
rials in the'rmodynapi cs are made in many 
segments. Each seqnent consists of a 
test which is co rres p9'nd i ng to the post 
test of a CAI segment' The midshipman 
who passes the test is considered to 
have completed the specific segment and 
achieved the desired objectives of the 
.segment. The testing segment requires 
the m.idshipmen to go through a certa>n 
sequence of questions and to answer cor- 
rectly each question po-sted in this se- 
quence. He IS told that he is wrong when 
an incorrect answer is picked However, 
the correct answer to a question is not 
provided in the test. The midshipman who 
fails the test, twice in a row, is given 
the 1 r?f o'rma t i on "cut off, and see your 
instructor. " The problem is either that 
the midshipnan needs some additional in- 
struction or there is an information gap 
in th^e testing segment. Some improvement 
to the*testing segment should be made. 



Di scuss 1 on 



A 



. Tutorial learning, especially under 
total imm.ersion is too ambitious con- 
.si^ering the present state of computer 
usa^e as an educational learning de^vice. 
Material appropriate for . computer-ass i s- 
ted testing seems* most effectively in 
the acea of training sucli as use of 
steam tables, gas tables, etc. , 
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'CI^EAH: ^ pfALisnc catc systw for u^se in publix schools 

^ , ' . . • ' ^e^nnetn Kayser, Data Frqcessing Manager ^ ^ 

*' **" • ,^ * . ^ Jeffrey i^lein, Matherratics Supervisor^ - 

. V " Kdr.ilapan*£ngl v.htijwn Pe'nunal Schools, tngl is^itown. New J^^y 077?6 

. ;n :-/;.*.'itif develoDnjeot a fiutricL-wide ^ nathematics curriculum was undertaken. Th'e approach 
' V' ^^^^ • ^'or ar^nievement -assrsStmpnT ^as criterion referertce testing. Students would be tested 
•?ri eich ..t^jf -"ttv"^ "usTrvg-test > tet^rs » taVen' frnrrl'drf item barjk based on the curriculum! objectives.^ Ao this end, 
ri c-'-r^i*V"^ oV. teat hers ►wa^ -con'r^irhone^'l to create such a bank pf'Tes^t questions br ite»T!s. The ne5^t 
;rfitu\ w'> the >ogi;^iKS of j)resent i^q-the ^est 'items to the .students . Tests containing groups of object- 
^,v<^--,v.!^.t./ wpre 1 ikely tO"t?e tesTe^ together were -created. Two dTfficulties ensued. Fir^t^ the cost 
*Vi,!ncVAf'>!*'>.ose tests pnnled, across all grade levels, v^as proJjibitive. Second, the objectives on a 
ridr ti'^j",}; test w«r* not 'necessarily the exSct ones a teacher might, want to test since the objectives 
•Tyjlrj t'»-.v,V.Mt*r different seauences depending on. the student, and teacher. This paper will^^QScrib^ 
^o^. 'tht^se V*\-bTen:^ v^re'solved by.&tili:ing a Computer Assistjetj Te^t Construction System »^a&ronymed^^ 
/•y^*-' L6"~P'jter iJe'fer^nced'fv3?uat>J'^^.^nti Assessment ir^>?dthenia.tics,^ye loped totally 'in house" fc 
Mu' jiStrMv'*::.^ jSf^ Systm/3-*ilodel 10. Bot'»r ihp ^ystem<; deSifgn and softwafe will tie described.^- ' 
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irrftOouCTic*'; 



et^in-, AfTuief-^ \o th^\que&<ion ^iHow t-tttnti^e 
■ ne's inst '*i'^:t:it>m{] 'prtSgrSm?" uSwaD >J proves .* 
rr^. -t rh''^jjnhl\ frU^V^ ting exporience'for 



L^hU fr*a,^ c/*ating^ 
■<i,Ai-<j t*i qh , ►adtPi n t s,t ra tors , s^upo r v i so '^s • 
if * ^ npi<^ alit*^'.!, Ttje Your-step pro*cedure' 
> .**lMpi tv'^Jow represervt^'V sif^Pt'^fjed ir(Od^^ vi^ich 
,.n !;>'*,jt i>T/ed wheri^'plarjrjVg a prograr; dPsv^ned 
> to at i»Ytwely. evaluate.' th^VinstructionaV . 
?ro';ri.T. of* t> givefi schpol ; \ " r\ " ' . , * 

] lie».'M:y the spec^^ic qbj^Cives of: ^^9 . 
insT * ucCionaJ program ' 
«ifdte evaluatioh mitrument^ t^at can ;t)& *.Vi^ed 
'o obtairi infonr^ation about- stii<i^nl Jchiovt?- 
r(*nt' ^\ related to the objettfves of the 
instructional arogram^ * ^ 

Qovelop and Vmj lenent proft'dures Cor ^ 
ad.pini<-.terinq he. -tests (or other evaluatu-^ .^7 
. instrunents ) * 

a. Collect data ar|d interpret results 

This pr^esentation will be concerned primarily 
with pciints 1 and 3- of the above mode\^ Hpwr. • ^ 
over, tie fore prc^^eedin^ *to these aspectSv^a. • * 
digression into the philosopKy under! ying^poifit ^ 
1 IS necessc^ry. . • * . /.\ 



The pros and cons 
the desirability 



RUCTIONAL 08J£CTtVFS 

concerning the contrbversy pv«t 
(of- unders^rabilityj j of 
delineating instrjctional objectives (also known 
as performance or 
bo » ei terated wit 
ion. The specif i 
lonal obje^ tives 



b^avioral objectives} will not 
in the* scope of this presentatn 
iCy and completeness of nnstruct- 
)as been a point of cjebatp for 



many years. Theoretically, a well written 



."^structional objective should state '( 1 )'what isV*- 
U that the student t/ho has n\as\ere(1 the concept) 
or skill identified in each objective v^^ll be'arbl^ 
to do, (2) under >/hcLt cor>dUions v^i 1 1 ^le be'^^ble 
to do this, an(f (3)'to what extent will^he bl? a61e 
to do this. 

'In practical usage /suffice it to say that. one 
[T.ajor advantage of an instructional objective^ 
IS derived*from the simple fact thai it is v/ritten. 
Once It IS written, it is visible. Once if'i'S 
visible. It can be reviewed, evalua'ted, nradlfied 
and improved. As the movertient for raccountabil ity 

*in education increases, this necessity for in- 
creased \isibil ity and specif icity'in objectives 
••wiM become apparent- No longejh'will education 
be able to c>oak inadequacies, incompetencies^ 
and mediocrity 'behind the guise of vague*ly st^ated\ 
*' unce^stiraye jgc»als ancl objectives. • • 

The rather* ot)vious .purf^ose of an 'instructional 
.'.'Objective should be to ma(je clea*" to teachers,^ 

Sttjdfint$ ajici'Dthejr interested person^ wKat students 
' should N jfble. to/do as a result of^ l6eJnslrucT- 
•* ibnal program, jjnfortunatjely, School tsy^tems ^ • 

cpiTijKjnIy lack. comprehensive and reasohably 

Consistent sc't; of. instructional objecUves. 

^/''lRIJER10N;REf;£ft£NCE0 HSTING 
• . * - * . • * 

*.*'••**•■ * 
•The'ruethod of ^eVdluatt<in commonly associated with 
• " in$.trirctional 'Objectives .is known as ^criterion- 
re'ferenced testing. Jn,t^is*mode Of evaluation-' 
(iTi c<2n*tr^>*t?td. n^nnrreferenced ^eslting^ , no 
attempt is to- compare tb'$ pdrforma*oce of an ^ 
, i^dj v^tJual w5th t'na^."^'^*^t}iec.$,/-.Ra*iherv one 
^ seeks to evalijato. pprfonnanc^ 'in/t^ms Ji^f whether 
*^n individual has^c^^eS^'ed *or.haV tai lec-itO achieve 
specific instr^io'pal objt^tiyes/* If .se]e^s'-to \ 
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answer the question, "What specific skills, 
knowledge, and understandings has a pupil, 
acquired?** In its (post elemental form, the 
response of each/^Jupil to each test item is 
evaluaied as correct or incorrect; the skill 
or ability measured by an individual test item 
has Either been mastered or it has not. 

Perhaps the most fundamental assumption 
implicit in criterion-referenced testing is 
that there exists a generally agreed upon 
hierarchy of skills or knowledge ia the field in 
which one w^nts to test. BasicalTy, the criterion- 
referenced approach is of little valbe unless the 
assumption u made that mastery of one skill or 
bit of knowJ^ge is essential- for mastering 
another skill Vr bit of knowledge' of a somewhat 
sinilar charactipr at a higher level of. difficulty 
0!'_ <^?p1exi ty. \ 4 , 

\ THE PROBLEM 

Based upon these precepts and phi losophies,' the 
• devolopmeft of a district-wide kindergarten 
through eighth grade mathematiq? curriculum was 
und;?rtaken. This object>ve bas^d curriculum, 
was to be developed in-house by a committee o^, 
teachers representing the various grade levels. 
The actual content of the' curriculum wa's to 
coinprised of a corapiTafion of generally acceptabJe 
mathematics objectives from yariou-s source^, , 
including textbooks and currently available 
listings 0^ objec^tives. These objectives yyou»ld 
thj&r be edi ted^iffd thodifiefi to best meet/the ni^eds 
of' our school district. * 4 * 

m - I 

This Objective b^sed^Tnathematjcs system was to- be 
Lomprised, of three components! (1) a/curriculum 
'guide which delineated the appropria^ instruct- 
ional objectsves foV each grade level, (2) a * 
record keeping component designed to chart and 
profile' individual and group progress through the 
specified objectives* and (3) a cr terfon- 
referenced evaluation component. Th(' first two 
compor>ent? of this system became operational and 
effective within twelve mo/Tths of their if^ception. . 
.The assessment component proved t^ be mln'e 
difficult to put into effective operational 
fonnat. , * 

The phi lo'sophy regarding the assessmeat component 
was it! concert with criterion-referenced testing. 
ToVJevelop the necessary test items for the 
objectives that were* developed during the previous 
year, a cQpsnittee of tethers was commissioned 
to create ^ bank of te^tr items or questions. These 

,lest items would therj/be used to test for student 
mastery of the speci'/ied objectives. After the 
te^t items were developed ♦ they were organized into 
packets or grtnjps of items which wer? considered 
tt) be most^likely to be tested at Jtih? same time. 
The original intent was to have thesle packets 
printed ,and -madei available for use Ifor all teachers 
with all ^heir students. The high icosts of 

'printing prevented, this particular strategy. In 
tiea of this procedure, tests were placed on ditto 
masters (ort a one-shot basis) and as many copies 

'of the tests* as possible were distributed to the 
te4t:t^'ers. Teacher reaction to these tests was 

^ufte positive. However, certain problems arose. > 
Most importantly, the objectives which were 



grouped for a particular test were not necessarily 
the exact ones a teacher might want to test since 
the objectives could be taught in different 
sequentes depending on the needs of the student. 
Secondly, a-fter the first year of use, the supply 
of test was depleted and were no longer available 
for teacher or stu4?nt use. 

The need for a more flexible and economically 
available method of reproducing the criterion- 
referenced test items was apparent. Flexibility- 
was necessary in terms of grouping test items, 
number of test items per Objective, number of ' 
test items per test, pre-tests and Interim 
assessment tests. A j^eadily'available method of 
reproducing the 'tests was necessary in terms of 
quantity of te*sts required, various puJ^DOses of « 
testing and economy of- reproduction. ' 

0 4 

It was to meet these needs and maintain the ' 
cr iter ion -referenced portion of the mathematics 
curriculum that CREAM, pcronym for Computer 
Referenced Evaluation and Assessment in Math- 
ematics, was developed. 

.DESIGN CRITERIA AND SYSJEM DESIGN , 

The design cri tenia for CRtAM was developed 
^und(5r the direction of the Mathematics-. Supervisor. 
The initial ^requirements of the system' were 
^ relatively simple: provide a*method for storage ' 
an^ retrieval of tifext-t^pe information. In 
add iti^fcj^sBveral /parameters for format of input 
and .00 ^ut were established. For example, it . 
was decided that only short answer type questions ^ 
would b6 used (multiple choice, fill-in, true- 
false). Maximum item (question) sizes were 
established as well as output formats. In the 
final (but tentative) design ♦the system would 
operate as follows.- For each grade level, a 
bank of iteiTjs would be established. This bank 
would contain approximately ten questions per , 
objective (roughly 100 objectives per grade) 
although. the actual number of items per 
objective could vary from objective tb objecti.ve. 
When a classroom teacher decided to test a class, 
group of students, or individual student, he/s)1e 
would submit a form specifying the grade level 
'and the number of items desired for each objective 



required e.g. three 
in objective 32, et< 
randomly. select fron 



items in objective 29-, seven 

The computer would then 
its item banks, iten^ .to 



fulfill the stated requirements and output a test 
on a 'ditto' master ready for reproduction., 
Minimum turn around time was an important design 
criteria. It. was also required that a teacher be 
able to retrieve alternate forms of the same type 
test. (different questions fof" the same objectives). 
The task of syst'ems design was then turned over 
to the Data Processing Manager. 

File slructure and access' techniques were the 
first k^ems considered. The restrictions imposed 
by the limited hardware (IBM System/3 Model 10) 
were of^ major consideration. After several 
iterations, it was decided that two files would be 
used .for the item bank. The primary, or main "ddta 
file wodld be direct random access and cdntain the 
actual item text. This file would be "mapped" by 
a second file which woul^ provide retrieval 
information for the main file. This procedure will 
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be described in more detail later. 

T^e next phase of design was the selection of 
a prograJiyning language. VORTi^AN was initially 
selected cjoe to the Ipgical type operations 
which were necessary in^item selection (random 
selection of a specified number of specific 
type items from a larger group) as -well as 
character string manipulation. However,' the 
mefficiencies of System/3 FORTRAN soon caused , 
the limits of our hardware to be exceeded (we 
were* working with 16K of main' storage) . The 
system'was t^en cojiverteji to RPG II (the Sy^tem/3's 
most efficient higti 1^1 language) and jt has 
evalved in that enyfronment. This choice did ^ 
pose some technpral problems, however*,, we were 
able to writey4me software without the Expense of 
a hardware u^rade. Specific techniqCes used in 
the coding will be described under "software 
design.: Thi se'lectipn of RPG II did offer some 
substantial advantages. Its ease of file access 
and Its facilities for table and array handling 
provided the ability to write the main test , 
generator .uti 1 izing less than 12K of main stor- 
age. The entire system will run on a Systenn/3 
(any model) with 16K of main storage, 96 print 
position printer and one 5440 type disk drive. It 
can, in fact, be run in 12K, however, the 
required overlays significantly downgrade system 
throughput. * 

' * SOFTWARE DESIGN 

As previous^ly .mentioned, two files are u^ed^V 
the bank-retrieval system. They are created as 
follows! The t^xt file (ITEMBANK) is loaded from 
a unit record devicfe. Each record, contain? the 
text of one item a/ong with the objecti ve /number , 
to;which the i tern /appl ies . The. record length of 
this file IS 190 fcytes. ^11 but five characters 
a^e aval laM£ fori text and control characters, 
(control charStti rs indicate format considerat- 
ions such as skt ) a line, suppress spacing, end 
of Item, etc.). After this file has been loaded, 
it^^ sorted int > ascending order by objective 
number. At this stage, the item map file 
(ITEKMAP) is crejted as follows. The main text 
file IS processep and the records per objective^ 
number are counted. A record is wrUten in 



ITEMMAP containipg the 
.,number of i temS 
1 ve, and the /'re(lat,i ve 
disk address of 



objective number 
how many Ttems I 
*and where to fir 



objecti ve number, the 
'ound ^n ITEMBANK for that object- 
record number" or relative 
he f^iHst item for that objective/ 
in ITEMBANK. Thjefefore'," by acces^sing a record y 
in ITEMMAP (whicp is Indexed Sef^uential with the' 
as the Key), n the system "knows" 
)re stored, for that objective 
d the first one. ' • 



The system evoUejd in^a mathematics environ- 
ment which is pjssibly the most difficult subject 
(along with chemistry) which could have been 
Chosen. This \% true due to the special signs 
and symbols as well as the use of exponents, 
subscripts and ^superscripts. To accommodate these 
fee^tures, a -type of "sub»' or mini language was 
constructed. In actuality, it is more, a format 
discipline than a language. Exponentiation, for 
example, is* achieved by, writing the base, skipp- 
ing a lino (*), and then writing the exponent 
a'fter counting the appropriate number of s^ace?. 



This has proved a most viablemethod of handling 
this problem. In this way, long hand addition. 
Subtraction, ^d multiplication examples can be 
constructed inUhe traditional column format and 
we can even create such symbols as long division 
(/ 



Returning to the item selection process, items 
are selectecf randomly from the group for the 
specified objective. The* random number geneyrator 
IS o/ the "power residue" typ5 and generates 
evenly "distributed oecimal numbers on the 
interval 0-1 (approximately).. The procedure for 
Item selection is as n^Uows. Let us assume that 
the request which thfe machine is currently 
servicing calls for three items from -objective 
number 37. After proper editing. has taken place, 
objecti ve .nuTaber 37's record is accessed in . 
ITEMMAP. The -system determines , from this lookup, 
the number of items which are contained in 
ITEMBANK for objective J7 (assume 10) and the 
relative record number^l^ the ^f irst item for 
objective 37, (assume S67)., The program now 
invokes the random number generator and retrieves 
a decimal number (8 places) on the interval 
0-M(assume .87.325951). (Actual ly,i the interval 
for*the random number is 0-. 9999999$ since 
only decimal positions are allocate(^). t^ext, 
the random number is multiplied by the number 
of items contained for the objective, the 
decimal is truncated^ and 1 is added to thjs 
product giving the item to be selected (.87325951^ 
X 10 = 8; 8+ 1 = 9 ie. select Uie ninth item 
. "for objective 37). The system now mast find the; 
position(relative record number) of this item. ' 
Well, it "knows" the position of the first 
item for object 37 (567). It simply adds S** to ^ 
this giving the position of the desired (ninth) 
record (567 + 8 = 575). (The reader may^ ponder 
the necessity of Adding J ,to the product of 
the random number and number of items and then 
subtracting 1 to find'the relative record num*ber. 
In actuality, these self cancelling steps are 
not performed by the machine and are included 
here for the sake of clarity.) Now that the , 
relative record number of the desired item is 
known, it can be quitkly retrieved. However, 
• fi.rst It is checked against a "memory" array to 
determine if it has been used already in this*run. 
If it has, the program returns to the random 
I number generator and computes a new item number. 
If it has not/been used in this run, .it is 
recorded in the ''memory"' array (so it will not 
be used again) ahd the text is then retrieved. 



All that, remains to be done is to format the 
selected item for.output and print it. This, is 
accomplished by simple character string 
manipulation as gijided by the imbedded control 
character?". Seveijal edi,it1ng considerations 
take place such as assii^ing that a given test 
item does not cause page overflow (having a 
question split ov0r two Ra^es may causi^ sufficient 
confusion to a primary grade youngster as t(f 
invalidate th^ item). The item is then printed 
and the system returns to select the next item 
(in the above example twtr more items from object- 
ive 37 must be selected. It will then process 
the other requested objectives if any), further 
details concerning the actual code may be 
*obtavned by contacting the authors. . 
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BELLS AND "whistles 

The procedures outlined above represent a basic 
CATC system. There are several enhancements 
that may be undertaken, based on the needs of the 
individual user. Some of these will be described 
in this section. 

1. Machine Scoring - The logical extension of 
machine gener,ated tests is machine scoring. 
The computer Can easily generate and store 
an answeTTiey Ton each test it generates. 
This keyn»may be used in conjunction^>with afi 
optical "h^ark reader to supply complete 
scaring and analysis services. 

2. I tem Validation - Once machine scoring has 
been implemented* analysis of the test items. • 
may be undertaken. The most popular 

Xype is the point-bisenal correlation 
techhique in which the validity of multiple 
choice questions may be established. This 
type of data is extremely useful in up- 
dating the item bank. 

3. Stu dent Progress Profile - Another off-shoot 
of machine scoring, students' progress 
through the curriculum may be charted and 
-fed back to the student, parents, and tea- 
cher on a periodic basis to aid in identify- 
ing strengths and weaknesses. <sA variety of 
similar types of computer manag^ instruct- 
ion techniques may be employed e.g. student 
recerd keeping, identification of curriculum 
deficiency areas, etc. 

4. Mu Ui-entrant Selection Criteria - In the 
system outlined in this paper, only two 
selection criteria were employed i.e* grade 

' level and objective. Mt»is possible how- 
ever^ to build a hierarchal ^t^ucture of 
selection criteria which rr^ay be epteved at 
any level. For example, let us assume that 
a course curriculum. is a continuum through 
four text books. A selection scheme could 
be establis^Ted such that a teacher might 
request a test in certain objectives . 
Alternately the request might be for a 
chapter or grew <?f chapters which may 
contain many objectives or perhaps a unit 
which contains several chapters, or a book 
which consists of many units. Finally, a- 
request for a test on t^e entire course (a 
fi^al exam) might contain questions on all 
four books for that course. In this 'case 
^ the hierarchy from lowest level to highest 
would-be ob jective within chapter within 
bopk within course. 

i tffl .G eneration - In the system <rf^scribed * . 
above,, eath item had to be individually 
developed. That .is, iff ten items wer^ ^ 

* desired for ,an objective which stated "the 
student shall be able (o add a pair of two 
digit numbers", theri t^n examples of the 
type 34 + 79 = .? would have to be written 

^ and enterefJ into the hank. Th'el reader will' 
f no doubt recognize thajt M the form of Ibe 

* item fx f y ^ ?) were {established, then* the 
computer CQuld rafrdomlfy supply values for x 

I and y. This would grcjatly ease ^the burden 
of item preparation as' well as provide an 
^ almost limitless number of items. 
6 • Other Dis c i£lj n e s - The techniques described 
I herefn can be applied to many other subject « 



^ireas. In fact, ftiany disciplines lend 
themselves far more easily to CATC than does 
mathematics; chiefly because special print 
characters and swnbols may be eliminated. 
Any subject whicK has quantifiable sub-areas 
(such as behavorial objectives)' is a 
potential candidate for CATC. 

OUTPUT 

The primary output from the system is of course, 
the generated test. This is presented to. the 
requesting teacher in the form of a "ditto" 
master or rexograph stencil. In one mode of 
operation, the systefn also prints £he answers 
to the questions inrthe right hand margin of 
the stencil. The teacher may then "run off" 
one copy of the test with answers, place a strip 
of tape ofer the answer column, and then 
reproduce the test for his/her students. An- 
other type of output consists of a "dump" of a 
section or entire contents of an item baok. 
This is used by teachers in determining whether 
9r not the system contains the type of quest- 
ions which he/she needs. In addition it serves 
as a vehicle for deleting, adding, or changing 
items in the bank. 

While not machine generated, another set of 
material is helpful. This co/tsists of a booklet 
of diagrams and illustrations. CREAM is not 
currently capable of outputing these types of 
graphics. The solution to this problem is~to 
reference an illustration or diagram in the test 
item and provide the student with the required . 
pages of the graphic^ booklet. This is extremely 
useful*in subjects such as map study, art, music, 
reading comprehension, etc. 

Figure 1 shows the coding of a test item for 
entry Into the system and its formated form-.after 
output. ' ' 

Figure 2 is a page of generated test output. Note 
that €he request parameters (number of questions 
and objectives with the first Jine being the 
number of questions* for the objective beneat-h^ 
It on the second line) are printed at the top' 
of each test. • 
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A MAN-MACHINE SYSTEM FOR l&PLDJRADING OF SHORT OUIZZfS 



A. Deatsman * ^ " 
Assistant Professor, Mathematics 
Ohio State University-Mansfield Campus 



ABSTRACT. Frequent short Quizzes provide valuable feedback for both teacher and students and also 
encourage students to keep up with homework- In a system devised by the author, students write quizzes 
on Dunch cards. The quizzes are graded by a human who Inspects the students' work and places each card 
in an appropriate stack. This allows grading of the students^ methods as well as answers and the 
Qivinq of partial credit- The sorted cctrds are computer processed. 

By taking over. all of the paper shuffTIng, marking, and recording 'of qrades, the computer makes 
possible human qraoi^m of students' complete work at a rate exceeding ten ouizzes per minute- This 
may be a desivable alternative to conventional machine gradinq- 



INTRODUCTION 

If one surveys controlled experimenta- 
tion in education, it soon becomes apparent 
that the vast majority of studies comparing 
one instructional procedure with another 
report no significant difference between 
them- Th«re is one striking exception. 
-Many experiments have. indicated that frequent 
quizzes are effective in improving student 
performance. 

Or- B*. C. Ourkee sumn»arized her exhaustive^ 
review of the literature on Quizzing by stating^ 
"Most studies appear to support frequent test- 
ing at both high scho.ol and-colVege level ^ 
and in a number of' subjects Including mathe-, ji 
matics-"^ Her o^n study done on colleae level j 
algebra students Hevealed that when students 
were given frequent Quizzes their performana 
was improved by a difference which was signi- 
ficant at the -05 level - 

These firidiri(|s are certainly consisteni 
with both common sense and psychology. We 
would expect that students iho^study re^ulafly 
and keep up vrith a course^ will *do better." 
Freouent quizzes should encourage this- They 
also should Provide valuabl^ feedBack to' ttfe 
students, allowing them to correct misconcep- . 
tions, discover weak areas, and focus attention 
on the most Important areas of a subject- Very 
short quizzes are sufficient to accomplish 
these objectives if they are freouent enoygh- 
If, for example, a student expects a quiz which 



i B.C- Ourkee, ."A^Study of the Effects of Three 
homework Procedures on Achievement in Col- 
lege Algebra" (unpublished Doctor's disser- 
tation, Arizona State University, 1972) ^ - 
p- 20. 



will be similar to a randomly selected home- 
work problem, he must do all of the homework 
to prepare for it- Of course it Is of no use 
copy another students' homework- 
✓ 

Even short qitizzes can take a lot of time 
to grade if one has very large numbers ^f 
students- Machine grading, however, has 
several disadvantages- It is difficult to make 
UP a good Quiz of the multiple choice or some 
other ty]^e which can be machine graded. Also, 
Quizzes are valuable feedback for the teacher 
as well as for the students.' It is important 
that the teacher actually see jlid analyze , 
students' written work, so that he can see what 
errors are being made and detide what sort of 
help his students need- Also, I feel .that most 
of my colleagues Would agree* that it is very 
desirable to givevPartial credit for correct^. 
pr<^dures which yield wrong answers due to 
mlnor^errors. This is (as far as I know) 
beyond the present capabilities of machine 
oradinq- 



TKe remainder of this 'paper describes a 



system which al 



Quizzes at a ve )y fast rate by having a computer 



take ov^r all 
fllna, marking. 



ows a teacher ta grade sho^^t 



the time-con$i»ning' paper shuf- 
and recording of cjrades. 



ORIGIN ANO»NATURE OF THE SYSTEM 

The svsteml'gy^ew out'pf efforts to increase 
efficiency in grading ouizzes- I was convinced 
that much" of thi time required to grade a stack 
of ouizzes was SPent in simply moving the papers 
around- Signifipnt additional timeA/as spent 
making marks on pe papers, computing scores, 
and writing the kcores on the papers^ 
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The first step In Improving on the situa- 
tion was to have the* students write their — 
quizzes on 3 by 5 cards. The cards were much 
easier to handle than papers j and their small 
size kept me from making the'ouizzes any lonaer 
than necessary. I could grade these ouizzes on 
cards fast enough that I became frustrated by 
the necessity of pausing to write the arade pn 

\each card. To save time in doing this I began 
dimply sorting the c^rds into stacks according 
to their scor^. ' When finished gradina a Quiz 
I would have a stack of A's, a stack of B's, 
and so forth. By this time it took me as much 

. time to record the grades as it took to grade 
the nuizzes. I soon realized that a computer 
could do the recording for me »dnd also compute 
summary statistics. 

Computer Processing has prov,ed invaluable 
for this type of quiz grading. For the last 
two years we have been using the grading <nethod 
at a small campus with no suitable computer 
facilities conveniently available. After pre- 
viously having used a computer, the use of 
human labor for this work was, in comparison, 
most unsatisfactory. 'A student helper recorded 
grades, and she was* not always available when 
it was most convenient for us. Consequently 
it was difficult to get good turnaround time. 
The. mere recording of the Quiz grades for our 
larqe sections of freshman algebra students was 
very tedious, and of course totals and averages 
had to* be computed by hand. 

We are now using the computer at a nearby 
technical school to process qrades. Initial 
costs were no higher than for the student 
helper, and we~expect them to decline with ^ 
increased use of the system to. the*POint where 
they are considerably lower than for human 
clerical help. 
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DETAILS ON USE OF THE SYSTEM 

The student ouiz cards are each punched 
with a student identification number. They 
are punched in reverse so that the number 
appears on the blank side of the card. Aj 
the~beginning of a coqrse each student in a 
class is assigned a number and given a supply 
of cards on which his nurnber has been punched. 
By qiving students only a few cards initially 
and bringing a box of additional cards to class 
each time a quiz was given we have had no , 
dificul'ty in qetting students to use correct 
cards in sufficiently qood condition for a ' 
card reader to process. 

Each student writes a ouiz on the blank 
side^if a card with his number punched in it. 
"To Oracle the ouizzes one simoly inspects each 
quiz card and decides what* grade is to be 
assiqned. Nothing need bB marked on the card. 
Each card is placed in an appropriate stack. 
This sorting process is very rapid. It is not 
difficult to qrade 100 short oufzzes in ten 
minutes. When the sorting is d)mplete a , 
"grade card" is placed on each stack which tells 
the computer what qrade to assiqhto the 
quizzes in that stack. The stacks aYe then 
assembled into a deck which is preceeded by a 
"ouiz card" indicating the number of the quiz. 
One or more of these ouiz decks make»up a deck 
which is nreceeded by a "clJss card" which 
identifies the class. An "end of class" card 
follows the deck. Finally, one or more of 
these class decks make up the input for the 
program. An optional "school card" identifying 
the school may preceed the rest of the input* 
and we use and "end of school" card to h^ing the 
program to an ordeVly termination. The 
following is a qraphical illustration of the 
deck'setuD for input: 



c^rds for other classes-*- 



j%nd of scho ol 



cards for additional quizzes from 
the same class as those preceeriinq 



^nd of class 




/ more qrade cards and student 
J_oui2 cards for the same ouiz 
as those preceedino 



students* ouiz cards »to be 
a^^sioned the orade indicated 
by the grade x:ard irtnediately 
Preceeding 



The printout from the computer shows each 
student's score on each qu1z to dat/». It also 
shows, the total number of points accumulated by 
.each student and his average and percenta.ge. 
We do not mark on students' qu1z cards or return 
them to the students, but rather go over each 
quiz as soon as the cards are handed 1n^ The 
students estimate the grade they expect to 
receive on the quiz and make a»note of It. ' 
Every few days we pass around a copy of the 
latest printout (on which students are Identi- 



fied by number only to preserve confidentiality 
of grades). ITa student does not fitid his grade 
to be ^very close to y^ha^ he expected he checks 
out the reason with tfie instructor. We keep the 
Quiz cards on file. until after the course Is 
over. We also, keep the grades stored on a disk 
file In case the cards might be lost and to 
avoid having to read In the same cards more than 
once. A sample of output from the program 
appears below: 
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Jhe program for the system Is oulte simple 
and straightforward. It allows* corrections to be 
made of graces on the disk file simply by Insert- 



ing an appropriate oulz card, grade cardi and 
student oulz card In the Input for the next run. 
A flowchart of the proqram.appears on the' fol- 
lowing, pages, and a CQBI)t"TTst^ ng Is available 
upon reouest for non-commerclaj use. 
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ABSTRACT: AL^COMBS is a comptiter system of test constructtpn' for any printer-pomposable question set re- 
trievable on any Boolean combinationipf unrestricted question classifications. Free-format entry of 



variable-length records is u&M^ed^n file construction and retrieval tor maximun system generality and 
ease of user access. Question cl-^lsif icatioj^ are designated in unlimited character strings without re 




erence to a fixed classif i^ajtion ^t or hierarchy, and as a consequence, questions can be classified wi 
qualif icatdons permitting multijJl^applications of any giv&n file. A user option providing for^exclusi 
of questions previf^sly offered a^ individual student or class allows for unique ro<*examination as re- 
|uired in»\:ompetency-based -instruction. . * ^ 



Introduction to Coi 



u^r-Assisted 



Test Qonstruction and ALLCOMBS |* 

t This *is a dlhscription d£' a «ysjtem of/ifcoputer- 
assisted tett construction undergoing itaplementa- 
tlon aC \exas Christian University. As is sug- 
gested by^ the system, ^r^vni, ALLCOMBS, this sys- 
tem offers some gener^ity in retrieval ar\d compo- 
sition oj^/ill combinations of questfoit types. 

Programs jof comp(it6r-as$isted te^t "Construc- 
tion, CATC, .have b£en under d^V^oprnent' for Vhe 
la%t deca(}e. Successf ^l^application^ of test con- . 
struction systems have been described at the Con- 
ference on Computers in the Undff graduate Curricu- 
*lum since the initial meeting pf t^is anni^al con- 
ference in 1970. Special interest Conferences on 
Computer-Assisted^ Test ConstrutClpn, meeting i*n 
1974 and 1975, *hav(^ "^f ^^^^d an increasing intfer- 
est in questions of^CATC implementation. 

Two {>rimary sources in the professional litera- 
ture oifqr overviews of the extensile acceptance of 
CATC in a diversity of Resting en^ljrdiytents. , Tfie 
first of these^ sources is the Masch 1^77 is&uc of 
the Journa'l Cjt Edtlcatlofial Xechnblogy w^^J^ is de- 
vo&ed to^ the topic^ of test cons true tion5f*^e roost* 
recent, com^rehtnsive source is the book Computer- * 
Assisted Test Construction edited by Gerald Ltppey 
and published b> Lducational Idchnology^ Publications 
in 19,74. ^ . ^ -Vj, 

The sumrvwry view offered here i?. that 'the effi- 
cacy of CATC systems has l>6en Established and tftat 
futther development offers the prospect of increas- 
ing appli9ability and e'fficlency. ^ The ALLCOMBS 
<iystcm to bo > presented here has evolved in this coh- 
toxt Qi CATC de\5elopmcnt as a generalized and flex- 
ible system of test construction explicitly designed 
to f4ij:tlttate jW'ady access to CATC by the' indivfdual 
user. « 
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DominanUdesign criteria for ALLCOMBS have been 
system generality*, flexibility,, and ease of user 
Access. The 'system is general in that it will com- 
pose all question format types such as multiple 
choice, true-fal6e, completion^ apd essay questions 
.retrieved, op the basis of any combination of ques- 
^Jlon classifications appropriate to the nee^s of 
the individual academi^c disc^ipHne or individual 
user. The system is flexible in that question for- 
. mat types and classifications may be expanded or 
contracted Selectively on an existing file and ret- 
'ord dimensions m^y be adjusted tn the spagifc require 
roents of given .system installSiion. Erfse of user 
^ access is proHded for by a ^fre^- format .sj 
•^le entry and retrieval, the e^parfsive> 
nostic (error) messages, an^ the accep/ 
verbal- syste^n parameters in file con^ruction and 
ret^^v^. ^ ' r f 

• There are ch^ces made in any system design 
*which amount* to trade-qf fs between desirable attrri- 

' butes which are .potential in the sysBem. fn the 
ALLCOMBS system, tjae election of Combinatorial gen- 
erality in specifitation of question classifica- 
tions forfeits certain »officiencies^f file organi- 
zation and retrieval whic!f* are fees (r able for very 
large files. At the other limit, the compositional 
generality of ALLCCHBS imposes a system of record 
tagging4.^searchtng, arfil editing for prfnt composi- 
t>ion whiclT^S uHf wilffeoessary. processing burden for ^ 
simpIified^ fixed-format printer reproduction of 

, exazninations<- v ^ 

^ * The ALLCOMBS system is appropriate \iHd effi- 
."^ctpnt for a large central range of awtl.^«tions •of 
^ computer-assisted test construction. Vjf|||xhin this 
*. range," there is sufficicijtt latitude tb.lpermit the 
/U9«r to. begin file devel^opment Pnd retrieval at'4 
. jtmplifitJd level which <^an extend in ti>phistlcation 
i^th the increasing .needs and experience of the 
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II. ALLCOMBS System Outline 



.ALLCOMBS is written In FORTRAN IV for batch- , * 
mode execution on cornputer systems utilizing 
EBCDIC character representation in words, 
the'hardward des^n of the IBM 360/370, the XeVox 
Slg^^a! 5/7/9, etc. Initial Input Is punch card 
to tape or disc flies and output Is by line printer. 

ALLCOMBS Is executed In one toode for question 
editing and file development and In another mqde 
for. question retrieval and examination composi- 
tion. Subprograms are shared by the two modes of ^ 
execution and rfre designed to be resident on the 
computer system disc file, or other Intermediate 
storage, subject toJfcall by the Individual user, 
ihe^e resident programs permit users to <ionstruct 
fiud access exclusive Individual or cociHon examlna|^ 
tio.i tiles. It Is not anticipated that assistance 

proiesslonal computer personnel will be required* 
u»r routine file development or retrieval with the 
system. • • ' ' 

Examination questions are prepared on p\inch, 
cards by the user for diagnostic editing and subse- 
quent^ transfer to ^magnetic t^pe or disc files. ^ 
Each block of records constituting a question ^ntry 
is composed of a permanent file number, a classifi- 
cation record, a question record^ an answer record, 
and a source recoud. The classification record is 
any set ot verbal, numeric or alphameric fields ^se- 
lected by the user for question classification, 
rhe quest><n^ecord l4,any combination of main p£u:- 
s^gpflphs of verSal text, response paragraphs; for 
multiple-choice questions and any literal character 
strings utilize^ for composition of blank response 
lines, tabular displays or line-printer gi^aphics. 
The answer r^Wrd can contain a fingle ch«racter 
designation .of a corre.ct response to a multiple- 
choice or true-false question and/or a full solu- 
tion or* explanation of the correct response to any 
question .type. The source record carries any appro- 
priate designation of the S(^rce of the qtjVst,ion in 
abbreviated or full bibliographic style.* ^ 

la the diagno'stic mode of .system execution, a 
trial composition is made of each question, the com- 
pletion of essential fields and records is verified, 
and storage limits for variable-length records are 
tested against parameters selected for the given sys- 
tem installation. The complete block of teco|fis for 
each question is printed in a hardcopy file to be 
.verified against the original source document.* 

After diagnostic editing, a given set of ^ues- ^ 
^tlons^s added to thq existing question ti'lc* File* 
* rc^rcnces to question olassif icatlons are exp&nded 
dynamically to include any new classif icatioRs in- 
troduced, the question and answer records are en- 
Jrypted'*for file security,'* and Lheu file is updated 
in all reapects for subsequent rdtrlevaf. 

access thcs^ question files, the user pre- 
parq,"^''an examinatton request which accesses^ ALLCOhBS 
1ft riirioVal and composttion .)«^e. In exAmlnatlon 
rctr/)2val 'an$I composition, the u^er may sciect^p-^,. 
tiorfs xn regflgrd to designation of -exajoinecs^ exawf* 
natiort content, and cxAmination compxjsit'ioft bt' lay- 



By appropriate reference to a separate file, 
examinees may be de^gnated as all students listed 
on a given class roster. Alternativery,> examina- 
tions can be prepared for students who are individ- 
ually named, or t:±ltiple examination copies .can be 
< '^prepared for students not designated by name. 

Content of the examinations may be selected by 
a combination of options which specify inclusion or' 
exclusion of particular questions or question clas^ 
sl'f ications. Additionally, the user may specify 
^exclusion of those questiins which have appeared 
previously on examinations given t6 a class or to 
individual students. ^ 

Individual questions may be designated for in- 
clusion or exclusion by referenc'te to the permanent 
file number of the question. The user may make this 
reference explicitly in the exaninatlpn .request by 
appropriate entry of the question* file numbers. Th^ 
\ separate file of the claes roster carries the re- 
• cord of previous ^amlnatlons by question file num- 
'ber and\selectlQ^^f an examination containing ai\ 
Tiew questions for students in the class makes im- 
plicit delusion of the questions which were used 
previously. 

The most general and flexible specification of ^. 
examination content is by question classification. *^ 
Examinations can be composed of any number of ques- ) 
tions "from any number of question'classd^f Ications t 
in any combination specifiable by the logical pr « 
Boolean operators .AND., .OR.., .NOT., and .EOR. 
(exclusive *'or"). Within the question sej^s select- 
ed by question classification, questions ^re se- 
lected randomly from those meeting criteria set 
in regard ^to these classifications and to the unique- 
ness of the "question set. The selection by 
tion classifiodtion can consist of a singl 
ication formingVthe set. from which random se 
is to be made, ofr- the logical expression can he a^ 
complex as may »e permitted by the FORTRAN compiler 
on the given comuter system.^ 

Options for con^^iXXonyot raVout Includ^ se- 
lect ion of tl^, number j5^M;(jestlon2|^ tS'be^ 
printed across a page of var lab I c^h^raw^n^ length., 
ALLCOMBS wtll print any feasible »pyiflcatlon of *• 
one through six ques.tion columns on a line of •22 ^ 
through 132 characters. Line breaks in the ques- 
^ tloj) text are generated automatically on the blank 
or hyphen ntsarest to the columnar^limit. A system 
default to full-page layout ensures that quc^tlon^ 
with lengthy literal strings wlll^be printed. 

As Implied ^n the previous description for ex- 
aminajiibn selection, the useV may ipecify a uniflue\ 
examination tor each student or a common examina- 
tion for all students. In examination coippo^ltlon, 
the user may ^ecif^ random reordering of questions 
^ on a cOtxDon exainination. i ^ ' ^ 

^An eKamlnatlott key i-s* pr6^arcd -for .erfch exanjtna- 
.tlol^ which, is unique in qu^stldn content or "brder. 
T&t £ cdniaon eXa^rlnation'^tn Identical or^fer' for all 
" sttidents^ the user may elect elthyar to pr.£nt one key 
for thc/coqoQn* examination or i'ndivl^ual copies of 
the key f6r*«ach*ex«mlnati$«i;'* - \ - ^ 

J y ...^ . , ' 

• Ars 'an. •^d(fi clonal *pt^An-6f composli'fon, the use^ 
mo/ print* unique Irvstrvctlons. on .eatrlti exam set. f 




III. Insroductton to Data Preparation for ALLC0M3S 1 

The organization of question files and examina- 
tion requests for ALLC0M3S is based on the subdivi- 
sion of data mtfo fields, records, and record blocks. 
U'ithin the question text, more direct referen6cs are 
nadc to th^cosiposition pf . these general subdivisions 
into question nain paragraphs, multiple-choice re- 
sponses or subparagraphs, and literal strings of 
characters. ? ^ 

Excepting the fixed field* designated far the 
pernvanent question file number to facilitate card 
handling, all fields, records, and record blocks 
are punched in free format with special characters 
used 10 denote data divisions. Data divisions may 
begin in any card position and entries 'nay continue 
without regard to the physi^cal break between succes- 
si^Je cirds. Column aighty of a given* card and -col- 
urm one of the succeeding card are interpreted as. 
adjacent puuch positions and the user nay, conceive 
of the input medium as a continuous tapffe. 

Within t\ese data divisions, literal records are 
utilised for those comj>^ositions foj^whTch automatic 
line breaks based on blank or hyphen Separation 
would be inappropriate. Literal records are uti- 
lized for such purposes as composition of tabular 
displays, reprpduction of line-printer graphics and 
statement of nvitheraatical expressions. Additionally, 
the user instructions for an examination are repro- 
duced Vrom a literal record without program editing, 

WitV the fcx^eption of ^ these literal records, ^ 
leading blanks are accepted within all data divi- 
sions and trailing blanks are accepted In al*l but 
numeric fiel(is. Given this system of free-format 
data preparation which accepts leading and trailing 
blanks, the user can organize data decks for maxi- 
mum personal convenience and readability. The 
principal obligation of £he user in preparation of 
question files and examination requests Is to ob- 
serve certa'ln minimum conventions of record desig- 
nation and ordering. 

, Excepting some more elabor^t^ tabular and 
graphic (Compositions, questrion files CypjLcally can 
be 'prepared directly from original written sources. 
Keypunch j^orps are neither required nor recommended. 
*^ The free -format conventions of ALLCOMbS petmit the 
user to %on^entfate on content rather than form. 

Procedur3% for preparation of question files 
and examination requests are presented by^example 
in the following two- sections. *The presentation is > 
expository and explicitly avo£<hs some formulisms of 
^ outline and definition more appropriate to a tech- 
nical reference do^urofent. <^ 

' * ' ' 9' 

* IV, PreparatJLog of ^ Question llecord Blocks 

> .Illustratloix I on the following page shows an 
cxdmlna.tion format which C9uld be obtained by se- 
lection o^ t^o coliimnf of ^questions to be printed ^ 
on a sevcpty-character lino. The questions have 
"been chosen to illus.trate the cotspositioh of 
ffiultiple-choice, completion, true-false ^d essay 
. questions. Que/itioni I and 2 also demonstrate the 
• 'application of literal ch<|racter ^ings in qucs^ 
tion cos;pdtition. 

• . * • f 



Illustration 2, shown on the next page following 
Illustration 1, shows card images of the four blocks 
of question records which could ^ave originated the 
question file used for the retrieval in Illustration 
1. Begirtning with th'e first line in Illustration 2, 
reading 75062804 ^ MULTIPLE CHOICE . . . , which 
is to be read as the first card in the deck, each 
print lin^ in Illustration 2 ^ep^fesents successive 
cards in the original card file for the questions. 

The four questions are identified by the four, 
eight-digit file numbers 75062804, 75123008, 
76022703, and 76022756. These file numbers may 6e 
any eigh&-digit set, but as used here represent the 
date-sequence of the question. The four pairs of 
digits in the file number denote the year, month, 
day and s&quence during the day of the quest'lon 
origination. Used in this way, the uniqueness of 
the file number is more nearly assured ^and the file 
number can provide a basis for file aging by origi- 
nation date. 

Each record in the question record block is 
t^erminated by a record mark; the record mark se- 
lected here is the ampersand, & . The first record 
in the block is the classification record w^ich may 
contain one or more verbal or codified classifica- 
tion fields beginning with the field mark, # , the' 
numeric symbol. The fi^st question classification 
record contains ^ MULTIPLE CHOICE FREQUENCY 
DIS'^IIBUTIONS # etc., and terminates with 
' ^ DIFFICULTY EASY ^ r 

\ \ 

The next record is t^t* (Question text, which 
Includes the responses to multiple-choice questions. 
The question record of the first question begins 
with # THE DISTRIBUTION OF . . . and terminates 
with ... BUT IS NOT GIVEN ABOVE. & . Main para- 
graphs of the question text begin with the field 
mark, ^ , the numeric symbol. 

Multiple-choice responses begin with the "at" 
symbol,. , which initiates program assignment of . 
the alphabetic sequence "A.", "B.", etc.\ The tabu- 
lac material of tWe first question is composed in 
, literal character strings marked at each end by the 
*. "xjot" symbol, -% . 

The criteria for selection of the symbols for 
data divisions are that the symbols be unique and 

. not b^^eeded within the. text to be placed on file. 
F8r any giveh system installation of ALLCOMBS^ the 

* user may make any choice of data-divisioti symbols 
which meet these criteria. 

In composition, a line ii skipped automatically 
before each main paragraph, before each subpara- 
graph used as a~multiple-choice response and be'fore 
each set of literal character strirfigs. l^nes arc 
"skipped", between literal character strh^g^ by 
, printing an empty string with the symbols: -i . 

The composition routfne centers the Pi,teral 
strings in any given question column ot\ the basis 
of the count of the longest string* in the ques'tion 
record and Bef t-justif ics -all other , literal charac- 
ter Strings to this consx>n origiti. If a columnar 
fbrmat has been selected which will not permit . 
printing of the longest string in a question, the 
compo»lti;on routine will dcfaul^t to the declared 
print limib of the system for printing that single 
question. 
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Illustration I 



I THE DISTRIBUTION OF TEST SCORES 2 A PERSONNEL WORKER IN A CERTAIN 
FOR AN EXAMINATION IS SHOUN AS: PLANT IS COLLECTING DATA ON 

EMPLOYEE ATTITUDES, SHE GIVES 
EXAMINAriON SCORES . . THE FOLIWING STATEMENT TO . 

EMPLOYEES AND ASKS THEM FOR A 
RANGE NUMBER RESPONSE FROM THE LISJ WHICH 

FOLLOWS. 

90 AND MORt 12 Jr> ^ 

' 80 THRU 89 17 ^ "I FEEL THAT I AM PERFORMING A 

70 THRU 79 32 VALUABLE SERVICE FOR SOCIETY 

60 THRU 69 24 *WHEN I DO MY JOB WELL." 

LESS THAN 60 15 ' ^ 

I - STRONGLY AGREE 
^ -rtiE FIRST DECILE OF THE SCORES 2 - AGREE 

IS? 3 - NO OPINION 

4 - DISAGREE , 

A. A SCORE OF 90. 5 - STRONGLY DISAGREE 

* 

B. TEN STUDENTS. ' COMPLETE THE FOLLOVfING: 

C. CANNOT BE DETgi^^INED" FROM VARUBLE NAME ^ 

THE DATA GIVEN. " ^ 

OBSERVATION UNIT 

D. CAN BE DETERMINED, BUT IS 

NOT GIVEN ABOVE. MEASUREMENT SCALE 

3 irirtrfck R^pER TO GRAPHICS FILE ' 4 IN USING SMALL SAMPLES TO TEST 

ITE^f'LC74/l82. ***** * THE DIFFERENCE BETWEEN MEANS 

OF TWO -NORMAL POPULATIONS, THE* 

AS SHOWN ON THE SCATTER SAMPLE VARIANCES aVe "POOLED'." 

DIAGRAMS FROM THE GRAPHICS * ) 

FILE,'tHB correlation BETWEEN EXPLAIN l^HY THIS PROCEDURE IS 

COLLEGE* GRADES AND SAT SCORES FOLLOWED AND WHAT OTHER TEST 

IS HtGHER THAN THE CORRELATION OF IfYPOTHESIS IS IMPLIED. ' 
' BETWEEN COLLEGE GRADES A?iD 

CLASS STANDING AS A HIGH ^ . (WRITE YOlJR ANSWER ON THE BACK 

SCHOOL SENIOR. . OF THIS SHEET AliD IDENTIFY IT 

BY QUESTION NUMBER.) . 

TRUE (OR) FALSE . * . 
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Illuscraclon 2 



75062804 # MULTIPLE CHOICE FREQUENCY DISTRIBUTIONS 4 DECILES JONES P 
ERSONAL FILE ^ DIFFICULTY EASY & 

4 THE DISTRIBUTION OF TEST SCORES FOR AN EXAMINATION IS SHOWN AS: 
lEXAhjlNATION SCORES^ -y 

T RANGE NUMBER-! -y ^ ' ' ' 

-i90 AND MORE 'I2-> 

^80 THRU. 89 17^,,^ 

-.70 THRU 79 32^ 

^60 THRU 69 24-. 

-iLESS THAN 60 I5i 

THE FIRST DECILE OF THE SCORES IS: (? A SCORE OF 90. Q TEN STUDENTS, 
3 CANNOT BE DETERMINED FROM THE DATA GIVEN. ^ CAN BE DETERMINED, BUT IS NOT G 
IVEN ABOVE. & ANSWER IS D. & JONES PERSONAL FILE 

75123008 ^ MEASURE^ENT AND SCALING # SHORT-ANSWER CO^IPLETION 4 DIFFICULTY MO 
DERATE' & -0 "* ' 

A, PERSONNEL WORKER' JN A CERTAIN PLANT IS COLLECTING DATA ON EMPLOYEE ATTITUDE 
,S. SHE GIVES TIE FOLLOWING STATEMENT TO EMPLOYEES AND ASKS THEM FOR A RESPONSE 
FROM THE LIST WHICH FOLLOWS. ?> "I FEEL THAT I AM PERFORMING A \JALUABLE SERVICE 

FOR SOCtETY WHEN I DO MY JOB WELL." 
-il - STRONGLY AGREE^ ^2 AGREEt t3 - NO OPINION^ ^4 - DI^GREEt -.5 - STRONG 
LY DISAGREE^ ^ COMPLETE TIE FOLLOWING: 

-I VARIABLE NAME t t -» ^ 

nOBSERVATION UNIT l.-y -y -y V 

-^MEASURE^ENT SCALE t. & 

VARIABLE NAME IS EMPLOYEE ATTITUDE. OBSERVATION UNIT l5 INDIVIDUAL EMPLOYEE. M 
EASURE>ENT SCALE IS ORDINAL. 4 
SU>^ffiRS AND PETERS CARD FILE^r 1973 & 

760i2703 ^ EXTERNAL GRAPHICS TRUE-FALSE ^ CORREUTION # DIFFICULTY EASY & 
^ irkirirff REFER TO GRAPHICS FILE ITEM LC74/182. ***** 

AS SHOwfl ON THE SCATTER DIAGRAMS FROM THE GRAPHICS FILE, THE CORRELATION BETWE 
EN COLLEGE GRADES AND SAT SCORES IS HIGHER THAN TIE CORRELATION BETWEEN COLLEGE* 
GRADES AND CLASS STANDING AS A HIGH SCH6oL S£NI0R. | 
TRUE & TCU ADMISSIONS OFFICE DATA FOR 1973 & 

76022756 # STUDENT'S T-TEST # DIFFICULTY "HARD # TjfeORETICAL ASSUMPTIONS OF 
TEST PROCEDURES # POOLING VARIANCES # ESSAY ANSWER # TESTS OF HYPOTHESES 
TESTS OF '^lEANS T^TS OP VARIANCES & 

# IN usiWG Small samples to test the difference between means of twoj^ormal pop 

ULATIONS, the SAMPLET VARIANCES ARE "POCJIIeD." ' , . ^ 

^ EXPLAIN WI^Y THIS PROCEDURE IS FOLLOWED AND WHAT OTHER TEST OF HYPOTHESIS IS I 
MPLIED. 

^ (WRITE YOUR ANSWER ON THE BACK OF THIS SHEET AND IDENTIFY IT BY QUESTION , NUMB , 
ER.) & 

Under tie test conditions specified, tie student t (or the f-ratio).test is us^d 

,'aND INVOLVES THE ASSUMPTION OF EQUAL POPULATION VARIANCES. POOLING OF SAMPLE 
variances is an AVERAGING PROCESS FOR THE ESTIH^ION OF THE COMMON POPUUTION VA 
RIANCE. THE TEST FOR DIFFERENCE BETWEEN MEANS UNDER TIESE CONDITIONS IMPLIES A - 
TEST FOR EQUAL POJ>ULATI0N VARIANCES, & 

EXAMINATION FILE FOR GRADUAT^QUALIFYING EXAM IN STATISTICS FIELD, & 
C t 
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The question rccorii can consist of any comblna'- 
cion of main paragrapl^ ^nd liccral chdracccr 
strings. Any numt>er oc number of mulciple-choicc 
subparagraphs may appear at the end of th« rcco*rd 
for printing at the bottom- of the question, .Vut tha 
number probably will not exceed five or six in ^ 
practipe. ' . • • 

The answer record,* printed on the hardcopy ques- 
tion tile and on the examination ke>', can consist 
of any character set. In the first questiAi in 
Illustration 2 on the preceding page, the tntry 
ANSWER IS D. ^ could have been followed bM an ex- 
planation of why this was an appropriate .cnoice or 
why the other choice^ were inappropriate. 

The final ^record is the* citation pf question 
source which is printed only on the hardcopy q^ies- 
tlou file. Hie source citation In the first ques- 
tion m Illustration 2 is duplicated by a qu&stion 
classlf icaiion which permits retriaval on tjie basis 
of the question source. A common application of a 
source ciiatiou in the question classifications 
would >^e clic composition of examiu.itions based on 
.a given textbook. 

N 

Finally, each record block is terminated by the 
cent svmbol, c . L'^e of an end-block character pre- 
vents the 'cumulat ion of record sequetxing errors 
from bne question to the rext in file construction. 

Question 2 illustratcs*thc use of literal char- 
acter strings for the unabiguous reproduction of 
the scale -Included in the question and for the»crea- 
tton ot response lines of minimum length to be used 
for answer completion. 

Question 3 illustrates i suitable flagging of 
an external £l le^ required for a questiq^. Addi-^^ 
tlonally, the question illustrates^ the use of the 
"conditional" symbol, I (EBCDIC hexadecimal to 
terminate the question record of a true-false ques- 
tion. Tills symbol causes automatic gcnerat^idn of 
t-he response line TRU^ (OR) FALSE . Alternatively, 
the user coul-d haVe entered^ <? TRUE FALSE in 
the question record as a multiple-choice response 
to facilitate automatic grading. 

Question U illustrates the fuUcrr use of the 
aaswer record for; the benefit of the grader or the 
exgipince. 

a . 

Control ot the editing and filing phase through^ 
ALLCOMBS is also specified with simplified verbal 
paramcter.s in free-format .^tyle. Creation of mag- 
netic files /necessarily involves some peculiarities 
of the job control language of a given installation 
and Ghis complete [topic of file construction is 
deferred until a reference system and instaLIation 
are defined. ' ' f 



V. Preparation of the £xamination Request 

Wliere the slash symbol, / , denotes the end of 
a Tccor^ and the cent symbol, c i denotes t^he end 
ct a record block, the examination request is made 
in the sequence of records shown in Illustration 3 
at the bottom of this page. 

As a specific example of the construction of an 
examination request, the four questions presented 
in Illustrations 1 and 2 in the preceding sectio,n 
could have been retrieved by the examination re- 
qi^st shown as card images in Illustration 4 the 
folbowing page. 

/Both the password and the- key for file encryp- 
tion and decryption are unique to each file. The 
pa*<Word is time dependent and 'ALLCOMBS will 
aciess the computer system clock to permK only "one 
aujthorized file entry in any time period specified. 
THe password disclosed. on ^one. submission of a batch 

un. cannot be used again for unauthorized. file 
access, nor is it likely that some trial process of 
password generation will* permit unauthorized file 
entry, particularly if the'decay interval of the 
password is kept short. File decryption is made to 
a scratch disc or tape only for those questions se- 
lected for the examination and the scratch file is 
cleared automatically after ^examination composition^ 

After the password in the examination reqoest, 
shown in Illustration A, the next three records con- 
tain identification of the instructor, class and 
examination. To facilitate .identification qf. the 
.output, the^irst twelve characters in each of <^ese 
tRree records are block printed on the title pag* 
for. the examination requesj. 

In the next record after- the examination iden- 
tification record in Illustration 4, ^^ptions have 
been entered for a seventy-character line to be 
-printed in two •columns for twenty 'recipients not 
named. The user has selected a common examination 
for all recipie9kts, with one key to be printed.. In 
this record, the user could 'have entered R0STER, 
rather than NOT NAMED » 20, to have obtained an 
exatriina^t Xon f^ r all class members listed on a class 
roster, or the u«er could have ^entered EACH NAMEp, 
to haye obtained examinations for students named in 
the last record blocR of this request* file, 

^or a common examination, the user could'fljkcc- ^ 
entered ^ SCfU\MBLE, to have the questions rearl^ed 
randomly*^ on each examination. If the examinees had 
been "selected by *ROSTER, or EACH NAMED, the user 
could have excluded questions previously given the- 
cl'&ss or individual student by enterli^g CLASS NEW, 
or STUDENT NEW^^ . - , ' 



JUustration 3 

■passworH / instructor identification / cUss identification / examination identification (by time 
or othfer distinction) / options for composition, for designation of the group of examinees, or ,for 
exclusion, of question* given oit previous examinations / lirteral instruction sot for this examina- 
tion / questions to be included or excluded by file number / questions to bo retrieved by question 
classification specified by Boolean operators / j • • then any number of classification retrieval 
records terminated. by the end-block symbol, c / identification of Individual student and questions 
on prior examinabions / . . . then any nuftiber of student records terminated by the end-block symbol. 
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PASSWORD / ,JOE 
ST^T 215^20 / 
PAGE WIDTH = 70, 
• THIS EXAMINATTOS 
OF EXAMmTIOp^ 
INCLUDE, 7602|756?S/ 
3, (- # >fULTIPLE CHOICE .OR. 
^ TRUE-FALSE 4 ) .AND. 



H 40NES - N'EELEY SCHOOL OF BUSINESS - XT 507 / 
FEB^.- DEMONSTRATION EXAMINATION / 

COLU>!NS « 2,.. NOT NiVhC-D » 20, COMMON, ONE KEY, 
HAS BEEN PREPARED TO DEMONSTRATE THE COMPOSITION 
B"E OFFERED IN THIS CI^SS THIS SEMESTER, c / 



SHORT-ANSWER COMPLETION ^ . OR . 
. NOT . # DIfFICULTY llARD 4- / c' 



Selections of options which would lead to proc- 
essing contradiction^. are signalled by diagnostix 
nfessagei; to the useV,'and irt instances in which 
reasonable r&soluCion seems possible, a system de- 
,faulc selection is made. 

The record following the options record holds 
the card image ii instructions to' be printed on 
each copy of the examination. In Illustration 4, 
this record begins with THIS EXAMINATION HAS . . . 
and' ends with the end -block symbol, . Following 
this instruction set is a record containing the 
designatioiV of'questions to be included or exclude^! 
by file number. The option INCLUDE, followed*by 
any set of file numbers will select those questions 
without regard to question classification. The 
option EXcLlDE, will omit the f.o I lowing questions 
by file numper, irrespective of any other selection 
option ex(j|ciscrf7~ 

The record beginning with the numeric field "3," 
indicates that three questions are to be selected by 
classifica&iohs which meet the truth criteria of the 
logiii^l expression shown in that^ record. Each ques- 
tion classification in this record begins and ends 
with the field mark, ^4 , the numeric symbol, and 
each logical op^ator ^s enclosed in periods, as in 
the syntax of FORTRAN IV. Parentheses control the 
ordering of evaluation of* the logical expressions. 

In retrieving with^ logical expressions of ques- 
tion classifications, ALLCOMBS passes through the 
complete question file testing for the trut^ of the 
logical expression in each classification sex of 
each question. Wher^ the classifications satisfy 
ttie logical^conditions specified, the question is 
copied ont9 a disc file to form a collection*of 
questions from which final, random selection can.be 
made. " 

tut examination request, by classification can 
consist of an unlimited block of these Classifica- 
tion rj&quest records^, egch having a specification 
of the number of questioos to be selected by a 
unique logicaT expression. The sample composition 
in Illustration 1-^ould have been retrieved by the 
record block, reading: . . . / I, ^IjWLTIPtE CHOICE<^ 
r I, #SH0RT-ANSWER COMPLETIONi? / I, ^TRUE-FALSE!? / 
,c / . ». . 



VI. 



Prospects for the ALLCOMBS System 



ALLCOMBS has been utilized in reduced form at 
Texas Christian University for exnmination\composi- 
tlon in a class organizcAi for s.clf-pnced instruc- 
tion. Major features oF the design have been veri- 
fied, but the full system it not yet implemented. 
*It is expected that the complete system will be 
. ' r . 
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iiqplemented and documented by September 1976. No 
data are available on the total computer system 
space requirements -or execution, time characteristics 
of ALLQOMBS. Nonetheless, some character istics^of 
the sy^teiD are sufficiently established that ^t does 
not seem prematura to offer some evaluation of them. 

ALLCOMBS will share ^ principal characteristic 
ot all CATC systems in freeing the instructor from 
major time demands of examination preparation. The 
instructor will be freed to consider alternative 
teaching forms such as self-paced, competency-based 
instruction. . , ' 

Designed, for the individual instructor, ALLCOMBS 
vill offer some convenience and simplicity of sys*tem 
access which should serve to encourage the use of 
CATC and the diffusion of its Advantages. 

♦ i 

A major distinction' of ALLCOMBS, ^which consti- 
tutes a major advantage of this system over some 
applications of 'CATC, Is the generality of the 
classification and retrieval system. It'wlU ac- 
cept virtually any system of olassificatio.n and 
will accept; expansion of the cla^sif i(fations as 
need and expeflence dictate. 

ALLCOMBS will accept contradictory classifica- 
tion sets for questions, so long as the classifica- 
tions are unique character set?. For exari^le, one 
user of a file may classify a question as DIFFICUL- 
TY HARD and another user may disagree and classify 
the question as RESCALE ^DDERATE. Retrieval can be 
made by reference to either classification without 
Interference to any other user of the question set.^ 
A generalized question file^can tonpaln classifica- 
tions vhich tailor subsets of the file to specific 
courses or instructor^. 

The generality of the format and symbolic desig- 
nation of data divisions should lead to. some effi^ 
Ciencies in exchange of question item banks. It 
should be possible to translate many item files in- 
to ALLCOMBS format with a minimum of editing by 
placement of symbols for data divisions in the ques- 
tion record blopk. 

And, finally, a note of disclaimer appropriate 
'to all CATC system*. The enjoyment of retrieval 
and compositional efficiencies ^comes after a suit- | 
able question file has* been created and verified: 
'The ds^xtfiopment and maintenance^ of a question item 
f Lli^requl^res a major cooxnitmcnt of resources in 
* drdqr to have a file of sufficient breadth and rep- 
lication to obtain full advantages of computer file 
management. Establishment of a question file .is 
ncft a problem to be minipiized. Ttie first (Question 
a potential CATC use]; must answer is whothdr the 
game is worth this candle. # 
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CTSS lil.\^SRiH)M I1.\C11LR SUPPORT SYSTFM) 



r.erald Lippey 
LUhsroom Support Systems 
San Jose, CA 95118 



CISS ua^ developed during 19b9 and 19 0 as 
afe'oxperimentul application under a joint 
■-tudv agreement between IBM and the Los 
\n>ieles Citv Unified School Distruct.. It 
m nov, in u'se at several locations, most 
01- them institutions of higher education. 

Ihi. application greV out of 'tha'recogn-it ion 
that a computer can perform an impoitant 
function bv relieving the teacher ot 
Clerical chores related to the construction 
of tests, while l(«u.-ing him free to make 
professional instruct lonal dec i-^ ions . 
Although it, has value in '™Pr°/""8 f 
cv' it is primarily a cost-effect ye tool 
fo; improving productivity.- /'o^-^. 
mat ion regarding the rationale ^hich un- 
derlies applicat^ions of this kind may be 
folmd in an article by the author ( 1975) . 

'crs's uses a centradiied data bank of u.ues- 
t ions on- disk storage to aid teachers in 
constructing. tests and exercises. The 
^^stem also scores students' answer sheets. 
It operates in bajeh mode, with teacher 
teouLsts, tests, and scoring reports trans- 
ported physically between the teacher and 
computer center. In this connection, it 
should be mentioned that the over- 
whel-img majority of educational insti- ^ 
Uptons are not sufficiently affluent to 
interactively administer tests in quan- 
^t5 or even to provide teachers with 
termrnals whicTi they might use to obtain 
te-^ts It would not, for example, he 
feasible for the Los Angeles schools to , 
provide CTSS service via tc?rn;>na s, a^d 
this economic barriei: is likely to remain 
for a long time. 

•CTSS assists teachers- in choosing ques- 
tions Banked item*^may be classified 

■ a onj several dim6ns?ons, .ncluding sub- 
tec? matter, difficulty, and behavior' 
ieSuired. They -are. ret rieve.d accord,.ing 
[ocUssification 'criteria ,set by the 
teachdr. Thus, a list of quest ions- day 
be'c'oAStruc<ed so that it conta^s for 

* examplV'a """ber oP questions <*> each 
of several toffies, with the desired mi^ 
of difficulty levels." Spec, ic items 
may also be requested by their ^Ident.fi- 
cation numbers. 



The Items selected are listed as an "ex- 
ercise," which is assigned !1 """^er and . 
remembered by the system, /h^^^f '^^^^.j. 
m-i-v specify that items be deleted or adU . 
ed! to produce a new "generation" of the 
exercise. This process may be repeated 
•for subsequent generations until the ex- 
ercise meets the teacher's nec^ds. 

Kith each exercise, the teacher receives 
the classification assigned to the items 
it contains. He also receives rpferences. 
\\ two sources of published material associ; 
ed with each question., A summary of the 
teacher's request and the . items ret rieved 
as a consequence accompanies the exercise. 

As a.teac-her option, an exercise may be 
p'rinted on reproduction masters. Also, 
SJon request, 'the system will create sev- 
eral "versions" of an exercise, each hav . , 
ing the items resequenced at random. 

When multiple-choice items are in use, /he 
correct answers to exercises are remember- 
ed b>' the system; so scoring keys netd not 
be. 'submitted. The teacher may add items 
of his own .to be scored and, in fact, may 
submi? answer sheets for tests entirely 
constS-ucted by him; in these cases, he 
"us? of course provide the "rrect answers 
to the questions he supplies^ Scoring of 
individual questions can b</ siipVessed. - 
The usual raw data, frequency di^ribu- 
<tions, summary statistics and lier^ re^ 
sponse breakdown are provided to the tea 
Cher. Questions "" be assorted o parts 
nf m exercise, each of which will be 
: ored s'eparatily.-If scran,bled versions 
were obtained , .additional reports for 
each version are available.. 

For each item collection, there is a" /tem 
statistics file which conta ins- usage data 
for eveJy question. This information in- 
cludes the number of times the question 
' was selected by the. system, deleted frofi- 
an exercise by a teachbr, and suppressed 
fjom being scored. The file 5lso coptains 
data o-btained from scoring, such as the 
cumulative numb6r of responses to each 
option aTid«a central tendency of discri 
^nation indices. A program is available 
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to'tot selected statistics aj^ainst spe- 
citicd thresholds to obtain a Irst of 
those questions likely to need rex-^isionf 
lliresholds can be set for high teacher 
rc%;ection rate, Iovn average discrirnina' 
lion index, unusual!) heav> use of a 
distractor, and very high'Or lo^ measur 
ed difficulty level/. 



.V 



Inlormation on system activit) accii- 
ruilated in a bX^ten statistics file. 
ihi> lile contains data on 25 different 
kinds of activity, representing the u^e 
of various LISS functions. The informa- 
tion Ks accumulated simultaneously over 
tuo time periods (long and ^hort). It is 
classified by item collection used and 

up to tv%el\e groups. to v^liich teachers 
r\.i\ be assigned. System activity reports 
r5.i\ bo obtained at anv time. 
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\n .issortment of additional programs is 
.a.nl.iblc for printing and updating item 
lujnks and other files.. Also, several do-" 
:en specialized programs were written to. 
.ill those who , oversee system operation. 

\cr a dozen collections of machine- read - 
.it U q^iostions have been prepared for use 
uilh LlSh>, and others arc imder develop-^ 
; 10 n t . 



Various aspects of CTSS have been des- 
ciibed elsewhere. Thcsje include the role 
It plays in the Los Angeles City Schools 
(Lippey et al, 1971) and the approach to 
Its implementation (Lippey, 1972). Since 
tliose were prepared,* several improvements 
i%ere made to the system. Inquiries re- 
garding Its present state and its avail- 
ability should be directed to the author. 

\Ithough LlSS appears to be th^most com- 
pr(;^ho^is 1 ve Uem retrieval and test printing 
system, there are a large number of com- 
puter programs -which assist with test 
construct lojn. The a^uthor has briefly re- 
viewed over 100 approaches and there 
mut;t be at least two to three times this 
numbei . Most of them make use of test 
Items in machine- readable form to print 
questions on demand. Many include, as 
'oes CTSS, computer assistance in select- 
'ing questions. Some "generate" similar 
questions using random techniques, and a 
fcfw assist only in the item selection , 
process, leqving the items and tests to 
be formatted and printed by conventional 
means. A fairly deta i led ^di scuss ion of 
cojuputer-assisted t.est ' construction can 
be found in the book by this title 
(Lippe)', 1974 ). 

References 

Lippey, G. A comprehensive question re- 
trieval application to serve cJ^asSroom 
teachers. In Proee'edings of the. 1972 Fall 
Jo int Computer Conte rcnce. 1972. 41. 655- 
ITST. > — ^ — 



9bH 



